[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:12:56 dist-test-slave-9hq9 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:12:56 dist-test-slave-9hq9 krb5kdc[399](info): set up 2 sockets
Sep 27 11:12:56 dist-test-slave-9hq9 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 11:12:57.034029 392 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:12:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:12:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:12:57.144577 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40841
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:12:57.221781 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:12:57.221961 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:12:57.221982 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:12:57.223084 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:12:57.223124 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:12:57.223138 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:12:57.223151 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:12:57.223163 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:12:57.224491 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:12:57.224660 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:12:57.224825 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:12:57.229645 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:12:57.229689 412 server_base.cc:1034] running on GCE node
W20260927 11:12:57.229657 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:12:57.229657 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:12:57.230206 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:12:57.230476 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:12:57.231617 412 hybrid_clock.cc:648] HybridClock initialized: now 1790507577231603 us; error 47 us; skew 500 ppm
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:12:57.233876 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:12:57.234361 412 webserver.cc:460] Webserver started at http://127.0.98.62:33557/ using document root <none> and password file <none>
I20260927 11:12:57.234597 412 fs_manager.cc:359] Metadata directory not provided
I20260927 11:12:57.234638 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:12:57.234771 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:12:57.235708 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "644d695251b44fb6ad4af816f4781b9b"
format_stamp: "Formatted at 2026-09-27 11:12:57 on dist-test-slave-9hq9"
server_key: "396166f2999c08ddd16d762e46e1af44"
server_key_iv: "c8cad887d35499f939c2a364a8342b7f"
server_key_version: "encryptionkey@0"
I20260927 11:12:57.236035 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "644d695251b44fb6ad4af816f4781b9b"
format_stamp: "Formatted at 2026-09-27 11:12:57 on dist-test-slave-9hq9"
server_key: "396166f2999c08ddd16d762e46e1af44"
server_key_iv: "c8cad887d35499f939c2a364a8342b7f"
server_key_version: "encryptionkey@0"
I20260927 11:12:57.237232 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:12:57.237958 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:57.238184 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:12:57.238281 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "644d695251b44fb6ad4af816f4781b9b"
format_stamp: "Formatted at 2026-09-27 11:12:57 on dist-test-slave-9hq9"
server_key: "396166f2999c08ddd16d762e46e1af44"
server_key_iv: "c8cad887d35499f939c2a364a8342b7f"
server_key_version: "encryptionkey@0"
I20260927 11:12:57.238350 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:12:57.252605 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:12:57.254484 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:12:57.254629 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:12:57.258579 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40841
I20260927 11:12:57.258639 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40841 every 8 connection(s)
I20260927 11:12:57.258940 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:12:57.259735 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:12:57.261837 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b: Bootstrap starting.
I20260927 11:12:57.262458 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b: Neither blocks nor log segments found. Creating new log.
I20260927 11:12:57.262732 480 log.cc:824] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b: Log is configured to *not* fsync() on all Append() calls
I20260927 11:12:57.263365 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b: No bootstrap required, opened a new log
I20260927 11:12:57.265136 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644d695251b44fb6ad4af816f4781b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40841 } }
I20260927 11:12:57.265265 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:12:57.265288 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644d695251b44fb6ad4af816f4781b9b, State: Initialized, Role: FOLLOWER
I20260927 11:12:57.265442 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644d695251b44fb6ad4af816f4781b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40841 } }
I20260927 11:12:57.265522 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:12:57.265545 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:12:57.265573 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:12:57.266122 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644d695251b44fb6ad4af816f4781b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40841 } }
I20260927 11:12:57.266213 480 leader_election.cc:302] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 644d695251b44fb6ad4af816f4781b9b; no voters:
I20260927 11:12:57.266330 480 leader_election.cc:288] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:12:57.266381 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:12:57.266533 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [term 1 LEADER]: Becoming Leader. State: Replica: 644d695251b44fb6ad4af816f4781b9b, State: Running, Role: LEADER
I20260927 11:12:57.266602 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:12:57.266614 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644d695251b44fb6ad4af816f4781b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40841 } }
I20260927 11:12:57.266994 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "644d695251b44fb6ad4af816f4781b9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644d695251b44fb6ad4af816f4781b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40841 } } }
I20260927 11:12:57.267131 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [sys.catalog]: This master's current role is: LEADER
I20260927 11:12:57.267124 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 644d695251b44fb6ad4af816f4781b9b. Latest consensus state: current_term: 1 leader_uuid: "644d695251b44fb6ad4af816f4781b9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644d695251b44fb6ad4af816f4781b9b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40841 } } }
I20260927 11:12:57.267225 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b [sys.catalog]: This master's current role is: LEADER
I20260927 11:12:57.267609 492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:12:57.268131 492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:12:57.268721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 412
I20260927 11:12:57.268821 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:12:57.269603 392 external_mini_cluster.cc:1428] Setting key 134b4cd8b3b622f7fb475c046ccb856e
I20260927 11:12:57.270884 492 catalog_manager.cc:1347] Generated new cluster ID: 13d4633752294816ad1177e31f2961ae
I20260927 11:12:57.270977 492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:12:57.283907 492 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:12:57.284564 492 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:12:57.287719 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:57.270873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.0.98.62:40841)
0927 11:12:57.274159 (+ 3286us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:57.274173 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:57.274205 (+ 32us) server_negotiation.cc:407] Connection header received
0927 11:12:57.274305 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:57.274311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:57.274359 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:57.274479 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:12:57.276430 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.278087 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.279539 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.280133 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.280227 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:57.284269 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:12:57.284273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:12:57.284273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:12:57.284283 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:12:57.285913 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:57.286386 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:57.286388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:57.286389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:57.286475 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:57.286730 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:57.286731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:57.286731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:57.286943 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:12:57.287103 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:57.287361 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 11:12:57.287490 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3246,"thread_start_us":80,"threads_started":1}
I20260927 11:12:57.293072 492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 644d695251b44fb6ad4af816f4781b9b: Generated new TSK 0
I20260927 11:12:57.293315 492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:12:57.324668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40841
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:12:57.416882 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:12:57.417092 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:12:57.417131 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:12:57.418239 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:12:57.418300 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:12:57.418345 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:12:57.419737 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40841
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:12:57.419962 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:12:57.420156 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:12:57.424858 506 server_base.cc:1034] running on GCE node
W20260927 11:12:57.424814 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:12:57.424830 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:12:57.424810 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:12:57.425177 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:12:57.425429 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:12:57.426595 506 hybrid_clock.cc:648] HybridClock initialized: now 1790507577426578 us; error 29 us; skew 500 ppm
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:12:57.428802 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:12:57.429193 506 webserver.cc:460] Webserver started at http://127.0.98.1:44215/ using document root <none> and password file <none>
I20260927 11:12:57.429397 506 fs_manager.cc:359] Metadata directory not provided
I20260927 11:12:57.429486 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:12:57.429631 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:12:57.430670 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "9ab7c320344c4abfb99e311c1e87de94"
format_stamp: "Formatted at 2026-09-27 11:12:57 on dist-test-slave-9hq9"
server_key: "bc0652369c1467236c205218bbd7fd1d"
server_key_iv: "06701f0cd2c711738963e570b4f8aea2"
server_key_version: "encryptionkey@0"
I20260927 11:12:57.431038 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9ab7c320344c4abfb99e311c1e87de94"
format_stamp: "Formatted at 2026-09-27 11:12:57 on dist-test-slave-9hq9"
server_key: "bc0652369c1467236c205218bbd7fd1d"
server_key_iv: "06701f0cd2c711738963e570b4f8aea2"
server_key_version: "encryptionkey@0"
I20260927 11:12:57.432281 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:12:57.432958 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:57.433115 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:12:57.433207 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "9ab7c320344c4abfb99e311c1e87de94"
format_stamp: "Formatted at 2026-09-27 11:12:57 on dist-test-slave-9hq9"
server_key: "bc0652369c1467236c205218bbd7fd1d"
server_key_iv: "06701f0cd2c711738963e570b4f8aea2"
server_key_version: "encryptionkey@0"
I20260927 11:12:57.433280 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:12:57.452039 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:12:57.453644 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:12:57.453744 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:12:57.453903 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:12:57.454190 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:12:57.454228 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:57.454248 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:12:57.454262 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:57.459894 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39373
I20260927 11:12:57.459967 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39373 every 8 connection(s)
I20260927 11:12:57.460261 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:12:57.468612 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:57.460912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57243 (local address 127.0.98.62:40841)
0927 11:12:57.460973 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:57.460974 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:57.461176 (+ 202us) server_negotiation.cc:407] Connection header received
0927 11:12:57.461591 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:57.461592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:57.461601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:57.461652 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:12:57.462553 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.463047 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.463975 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.464188 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.464249 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:57.466137 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:12:57.466140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:12:57.466140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:12:57.466150 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:12:57.467426 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:57.467802 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:57.467821 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:57.467821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:57.467872 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:57.468074 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:57.468074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:57.468074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:57.468212 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:12:57.468262 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:57.468490 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 11:12:57.468552 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 11:12:57.468816 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:57.460892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40841 (local address 127.0.98.1:57243)
0927 11:12:57.461034 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 11:12:57.461070 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:12:57.461509 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:12:57.461700 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:12:57.461703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:12:57.462300 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:12:57.462478 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:57.462483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.463123 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.463123 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:57.463876 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:57.463878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.464262 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.464263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:57.464355 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:57.464826 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:12:57.464833 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:12:57.466013 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:12:57.467510 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:12:57.467515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:12:57.467516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:12:57.467699 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:12:57.467929 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:12:57.467929 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:12:57.467930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:12:57.468020 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:12:57.468266 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:12:57.468270 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:12:57.468370 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 11:12:57.468459 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 11:12:57.468564 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260927 11:12:57.469199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 506
I20260927 11:12:57.469311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:12:57.469429 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40841
I20260927 11:12:57.469477 392 external_mini_cluster.cc:1428] Setting key 962c781cb63e4d09460a783291fdd737
I20260927 11:12:57.469544 635 heartbeater.cc:464] Registering TS with master...
I20260927 11:12:57.469735 635 heartbeater.cc:511] Master 127.0.98.62:40841 requested a full tablet report, sending...
I20260927 11:12:57.470309 444 ts_manager.cc:194] Registered new tserver with Master: 9ab7c320344c4abfb99e311c1e87de94 (127.0.98.1:39373)
I20260927 11:12:57.470881 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57243
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:12:57.490813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40841
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:12:57.567492 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:12:57.567691 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:12:57.567713 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:12:57.568852 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:12:57.568889 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:12:57.568917 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:12:57.570256 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40841
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:12:57.570425 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:12:57.570590 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:12:57.575145 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:12:57.575101 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:12:57.575322 640 server_base.cc:1034] running on GCE node
W20260927 11:12:57.575121 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:12:57.575574 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:12:57.575806 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:12:57.576960 640 hybrid_clock.cc:648] HybridClock initialized: now 1790507577576952 us; error 46 us; skew 500 ppm
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:12:57.579262 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:12:57.579660 640 webserver.cc:460] Webserver started at http://127.0.98.2:44309/ using document root <none> and password file <none>
I20260927 11:12:57.579878 640 fs_manager.cc:359] Metadata directory not provided
I20260927 11:12:57.579943 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:12:57.580090 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:12:57.581012 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "22d155be5aa048679bfed4e44002894f"
format_stamp: "Formatted at 2026-09-27 11:12:57 on dist-test-slave-9hq9"
server_key: "2e4a8ba8d0bb2cfdb2c02bad0485454f"
server_key_iv: "a5c485f23af40ceaa1edeb403fb85ade"
server_key_version: "encryptionkey@0"
I20260927 11:12:57.581357 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "22d155be5aa048679bfed4e44002894f"
format_stamp: "Formatted at 2026-09-27 11:12:57 on dist-test-slave-9hq9"
server_key: "2e4a8ba8d0bb2cfdb2c02bad0485454f"
server_key_iv: "a5c485f23af40ceaa1edeb403fb85ade"
server_key_version: "encryptionkey@0"
I20260927 11:12:57.582625 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:12:57.583294 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:57.583451 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:12:57.583542 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "22d155be5aa048679bfed4e44002894f"
format_stamp: "Formatted at 2026-09-27 11:12:57 on dist-test-slave-9hq9"
server_key: "2e4a8ba8d0bb2cfdb2c02bad0485454f"
server_key_iv: "a5c485f23af40ceaa1edeb403fb85ade"
server_key_version: "encryptionkey@0"
I20260927 11:12:57.583619 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:12:57.603322 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:12:57.605396 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:12:57.605556 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:12:57.605748 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:12:57.606015 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:12:57.606097 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:57.606139 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:12:57.606153 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:57.611917 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37937
I20260927 11:12:57.611976 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37937 every 8 connection(s)
I20260927 11:12:57.612247 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:12:57.614789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 640
I20260927 11:12:57.614888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:12:57.615034 392 external_mini_cluster.cc:1428] Setting key 0460a182fa9106d798ea01872eaf6f65
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:12:57.621101 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:57.612856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45403 (local address 127.0.98.62:40841)
0927 11:12:57.612937 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:57.612939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:57.613084 (+ 145us) server_negotiation.cc:407] Connection header received
0927 11:12:57.613480 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:57.613480 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:57.613491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:57.613541 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:12:57.614438 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.614954 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.616100 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.616287 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.616361 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:57.618387 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:12:57.618389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:12:57.618389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:12:57.618395 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:12:57.619816 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:57.620188 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:57.620189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:57.620190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:57.620246 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:57.620484 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:57.620485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:57.620486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:57.620599 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:12:57.620671 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:57.620935 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 11:12:57.621016 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 11:12:57.621263 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:57.612809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40841 (local address 127.0.98.2:45403)
0927 11:12:57.612981 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 11:12:57.613011 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 11:12:57.613403 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:12:57.613588 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:12:57.613591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:12:57.614176 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:12:57.614361 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:57.614365 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.615052 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.615053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:57.616017 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:57.616019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.616369 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.616371 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:57.616491 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:57.616951 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:12:57.616958 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:12:57.618288 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:12:57.619898 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:12:57.619904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:12:57.619906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:12:57.620113 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:12:57.620314 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:12:57.620315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:12:57.620315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:12:57.620420 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:12:57.620662 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:12:57.620668 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:12:57.620797 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 11:12:57.620911 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 11:12:57.621052 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":85,"threads_started":1}
I20260927 11:12:57.621690 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40841
I20260927 11:12:57.621784 769 heartbeater.cc:464] Registering TS with master...
I20260927 11:12:57.621965 769 heartbeater.cc:511] Master 127.0.98.62:40841 requested a full tablet report, sending...
I20260927 11:12:57.622373 444 ts_manager.cc:194] Registered new tserver with Master: 22d155be5aa048679bfed4e44002894f (127.0.98.2:37937)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 11:12:57.622795 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:12:57.638154 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40841
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:12:57.718583 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:12:57.718768 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:12:57.718791 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:12:57.719882 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:12:57.719915 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:12:57.719940 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:12:57.721232 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40841
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:12:57.721401 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:12:57.721563 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:12:57.726227 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:12:57.726238 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:12:57.726234 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:12:57.726426 774 server_base.cc:1034] running on GCE node
I20260927 11:12:57.726632 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:12:57.726857 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:12:57.728014 774 hybrid_clock.cc:648] HybridClock initialized: now 1790507577728005 us; error 38 us; skew 500 ppm
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:12:57.730306 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:12:57.730647 774 webserver.cc:460] Webserver started at http://127.0.98.3:35357/ using document root <none> and password file <none>
I20260927 11:12:57.730803 774 fs_manager.cc:359] Metadata directory not provided
I20260927 11:12:57.730839 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:12:57.730921 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:12:57.731839 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "0f9f23ffa8f6435a97db16c38f6a6532"
format_stamp: "Formatted at 2026-09-27 11:12:57 on dist-test-slave-9hq9"
server_key: "278be7d4d1c3c2ca032a635ee4e0be13"
server_key_iv: "8163768569bb65890d8601a07d23c914"
server_key_version: "encryptionkey@0"
I20260927 11:12:57.732134 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0f9f23ffa8f6435a97db16c38f6a6532"
format_stamp: "Formatted at 2026-09-27 11:12:57 on dist-test-slave-9hq9"
server_key: "278be7d4d1c3c2ca032a635ee4e0be13"
server_key_iv: "8163768569bb65890d8601a07d23c914"
server_key_version: "encryptionkey@0"
I20260927 11:12:57.733376 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:12:57.733984 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:57.734246 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:12:57.734352 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "0f9f23ffa8f6435a97db16c38f6a6532"
format_stamp: "Formatted at 2026-09-27 11:12:57 on dist-test-slave-9hq9"
server_key: "278be7d4d1c3c2ca032a635ee4e0be13"
server_key_iv: "8163768569bb65890d8601a07d23c914"
server_key_version: "encryptionkey@0"
I20260927 11:12:57.734427 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:12:57.755600 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:12:57.757292 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:12:57.757450 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:12:57.757665 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:12:57.757959 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:12:57.758013 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:57.758054 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:12:57.758132 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:57.764387 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40029
I20260927 11:12:57.764478 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40029 every 8 connection(s)
I20260927 11:12:57.764755 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:12:57.772490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 774
I20260927 11:12:57.772589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:12:57.772758 392 external_mini_cluster.cc:1428] Setting key 0da1cdfefbe9e8e029004974ceca9439
I20260927 11:12:57.773106 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:57.765397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45139 (local address 127.0.98.62:40841)
0927 11:12:57.765478 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:57.765480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:57.765639 (+ 159us) server_negotiation.cc:407] Connection header received
0927 11:12:57.766109 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:57.766110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:57.766122 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:57.766173 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:12:57.767161 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.767688 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.768671 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.768847 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.768903 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:57.770731 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:12:57.770732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:12:57.770732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:12:57.770738 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:12:57.771824 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:57.772145 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:57.772146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:57.772147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:57.772190 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:57.772380 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:57.772381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:57.772382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:57.772512 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:12:57.772584 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:57.772924 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 11:12:57.772986 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 11:12:57.773272 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:57.765357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40841 (local address 127.0.98.3:45139)
0927 11:12:57.765524 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 11:12:57.765557 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 11:12:57.765994 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:12:57.766229 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:12:57.766232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:12:57.766852 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:12:57.767080 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:57.767086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.767773 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.767774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:57.768597 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:57.768599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.768917 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.768918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:57.769022 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:57.769402 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:12:57.769409 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:12:57.770657 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:12:57.771878 (+ 1221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:12:57.771883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:12:57.771885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:12:57.772084 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:12:57.772228 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:12:57.772228 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:12:57.772228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:12:57.772304 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:12:57.772615 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:12:57.772622 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:12:57.772775 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 11:12:57.772883 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 11:12:57.773050 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260927 11:12:57.773700 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40841
I20260927 11:12:57.773782 903 heartbeater.cc:464] Registering TS with master...
I20260927 11:12:57.773962 903 heartbeater.cc:511] Master 127.0.98.62:40841 requested a full tablet report, sending...
I20260927 11:12:57.774333 443 ts_manager.cc:194] Registered new tserver with Master: 0f9f23ffa8f6435a97db16c38f6a6532 (127.0.98.3:40029)
I20260927 11:12:57.774704 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45139
I20260927 11:12:57.784605 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507577, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:12:57.805392 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:57.798704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49940 (local address 127.0.98.1:39373)
0927 11:12:57.798869 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:57.798872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:57.798882 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:12:57.798902 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:57.798903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:57.798915 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:57.798976 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:12:57.799445 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.800078 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.800708 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.800975 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.801023 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:57.803017 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:12:57.803023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:12:57.803023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:12:57.803028 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:12:57.804410 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:57.804689 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:57.804691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:57.804692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:57.804739 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:57.804880 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:57.804881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:57.804881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:57.804977 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:12:57.805066 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:57.805160 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 11:12:57.805258 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
W20260927 11:12:57.805721 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49940
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:12:57.826143 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:57.819884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49952 (local address 127.0.98.1:39373)
0927 11:12:57.819945 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:57.819946 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:57.819981 (+ 35us) server_negotiation.cc:407] Connection header received
0927 11:12:57.820017 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:57.820018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:57.820026 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:57.820104 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:12:57.820567 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.821020 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.821744 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:57.821929 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:57.821976 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:57.823981 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:12:57.823982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:12:57.823983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:12:57.823988 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:12:57.825294 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:57.825556 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:57.825557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:57.825557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:57.825600 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:57.825730 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:57.825731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:57.825731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:57.825798 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:12:57.825873 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:57.825945 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 11:12:57.825995 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 11:12:57.826625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 506
I20260927 11:12:57.831601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 640
I20260927 11:12:57.837038 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 774
I20260927 11:12:57.840929 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 412
2026-09-27T11:12:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1824 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[917](info): set up 2 sockets
Sep 27 11:12:57 dist-test-slave-9hq9 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 11:12:58.849452 392 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:12:58 dist-test-slave-9hq9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:12:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:12:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:12:58.954546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39183
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:12:59.030797 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:12:59.030975 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:12:59.030995 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:12:59.032109 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:12:59.032148 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:12:59.032163 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:12:59.032217 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:12:59.032246 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:12:59.033676 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:12:59.033908 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:12:59.034147 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:12:59.038683 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:12:59.038698 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:12:59.038802 930 server_base.cc:1034] running on GCE node
W20260927 11:12:59.038697 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:12:59.039215 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:12:59.039566 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:12:59.040721 930 hybrid_clock.cc:648] HybridClock initialized: now 1790507579040707 us; error 26 us; skew 500 ppm
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:12:59.043036 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:12:59.043396 930 webserver.cc:460] Webserver started at http://127.0.98.62:44685/ using document root <none> and password file <none>
I20260927 11:12:59.043561 930 fs_manager.cc:359] Metadata directory not provided
I20260927 11:12:59.043598 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:12:59.043677 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:12:59.044588 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "d3a7785ee1c24c6ea6fc31664545173f"
format_stamp: "Formatted at 2026-09-27 11:12:59 on dist-test-slave-9hq9"
server_key: "1c03a561a153b9204d20da8401c7e4d0"
server_key_iv: "567b139c5fc1fdd6e889372555abb55d"
server_key_version: "encryptionkey@0"
I20260927 11:12:59.044871 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "d3a7785ee1c24c6ea6fc31664545173f"
format_stamp: "Formatted at 2026-09-27 11:12:59 on dist-test-slave-9hq9"
server_key: "1c03a561a153b9204d20da8401c7e4d0"
server_key_iv: "567b139c5fc1fdd6e889372555abb55d"
server_key_version: "encryptionkey@0"
I20260927 11:12:59.046031 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:12:59.046979 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:59.047204 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:12:59.047284 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "d3a7785ee1c24c6ea6fc31664545173f"
format_stamp: "Formatted at 2026-09-27 11:12:59 on dist-test-slave-9hq9"
server_key: "1c03a561a153b9204d20da8401c7e4d0"
server_key_iv: "567b139c5fc1fdd6e889372555abb55d"
server_key_version: "encryptionkey@0"
I20260927 11:12:59.047380 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:12:59.062662 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:12:59.064251 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:12:59.064399 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:12:59.068198 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39183
I20260927 11:12:59.068262 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39183 every 8 connection(s)
I20260927 11:12:59.068580 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:12:59.069367 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:12:59.071588 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f: Bootstrap starting.
I20260927 11:12:59.072152 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f: Neither blocks nor log segments found. Creating new log.
I20260927 11:12:59.072446 998 log.cc:824] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f: Log is configured to *not* fsync() on all Append() calls
I20260927 11:12:59.073094 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f: No bootstrap required, opened a new log
I20260927 11:12:59.074669 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a7785ee1c24c6ea6fc31664545173f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39183 } }
I20260927 11:12:59.074797 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:12:59.074818 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a7785ee1c24c6ea6fc31664545173f, State: Initialized, Role: FOLLOWER
I20260927 11:12:59.074992 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a7785ee1c24c6ea6fc31664545173f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39183 } }
I20260927 11:12:59.075052 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:12:59.075093 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:12:59.075140 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:12:59.075631 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a7785ee1c24c6ea6fc31664545173f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39183 } }
I20260927 11:12:59.075736 998 leader_election.cc:302] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3a7785ee1c24c6ea6fc31664545173f; no voters:
I20260927 11:12:59.075924 998 leader_election.cc:288] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:12:59.075968 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:12:59.076169 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:12:59.076288 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [term 1 LEADER]: Becoming Leader. State: Replica: d3a7785ee1c24c6ea6fc31664545173f, State: Running, Role: LEADER
I20260927 11:12:59.076373 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a7785ee1c24c6ea6fc31664545173f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39183 } }
I20260927 11:12:59.076736 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3a7785ee1c24c6ea6fc31664545173f. Latest consensus state: current_term: 1 leader_uuid: "d3a7785ee1c24c6ea6fc31664545173f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a7785ee1c24c6ea6fc31664545173f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39183 } } }
I20260927 11:12:59.076743 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3a7785ee1c24c6ea6fc31664545173f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a7785ee1c24c6ea6fc31664545173f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39183 } } }
I20260927 11:12:59.076844 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [sys.catalog]: This master's current role is: LEADER
I20260927 11:12:59.077005 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f [sys.catalog]: This master's current role is: LEADER
I20260927 11:12:59.077087 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:12:59.077564 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:12:59.078655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 930
I20260927 11:12:59.078747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:12:59.078903 392 external_mini_cluster.cc:1428] Setting key 36298f4b8b79930a670af0ae2bedcefa
I20260927 11:12:59.079072 1010 catalog_manager.cc:1347] Generated new cluster ID: 207a5f8618de41d3858401db81e268ce
I20260927 11:12:59.079125 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:12:59.090018 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.079453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37986 (local address 127.0.98.62:39183)
0927 11:12:59.079580 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:59.079588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:59.079615 (+ 27us) server_negotiation.cc:407] Connection header received
0927 11:12:59.079718 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:59.079724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:59.079776 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:59.079908 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:12:59.080651 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.082014 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.083093 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.083667 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.083759 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.086514 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:12:59.086518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:12:59.086518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:12:59.086530 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:12:59.088448 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.088819 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.088822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.088822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.088926 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.089117 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.089118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.089118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.089443 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:12:59.089596 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:59.089660 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 11:12:59.089788 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260927 11:12:59.129752 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:12:59.130343 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:12:59.137981 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3a7785ee1c24c6ea6fc31664545173f: Generated new TSK 0
I20260927 11:12:59.138170 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:12:59.166844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39183
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:12:59.245646 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:12:59.245823 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:12:59.245846 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:12:59.246924 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:12:59.246958 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:12:59.246984 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:12:59.248276 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:12:59.248435 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:12:59.248584 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:12:59.253332 1024 server_base.cc:1034] running on GCE node
W20260927 11:12:59.253317 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:12:59.253316 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:12:59.253351 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:12:59.253674 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:12:59.253937 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:12:59.255070 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790507579255056 us; error 42 us; skew 500 ppm
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:12:59.257352 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:12:59.257743 1024 webserver.cc:460] Webserver started at http://127.0.98.1:38967/ using document root <none> and password file <none>
I20260927 11:12:59.257946 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 11:12:59.257987 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:12:59.258144 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:12:59.259090 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "2f3fc5db13004caeb377884dac3b34c9"
format_stamp: "Formatted at 2026-09-27 11:12:59 on dist-test-slave-9hq9"
server_key: "085018c4bd61adfb1e2d3b936f65a60d"
server_key_iv: "e4d45150f22f5cc50e906022e99144a3"
server_key_version: "encryptionkey@0"
I20260927 11:12:59.259423 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2f3fc5db13004caeb377884dac3b34c9"
format_stamp: "Formatted at 2026-09-27 11:12:59 on dist-test-slave-9hq9"
server_key: "085018c4bd61adfb1e2d3b936f65a60d"
server_key_iv: "e4d45150f22f5cc50e906022e99144a3"
server_key_version: "encryptionkey@0"
I20260927 11:12:59.260679 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:12:59.261335 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:59.261502 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:12:59.261583 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "2f3fc5db13004caeb377884dac3b34c9"
format_stamp: "Formatted at 2026-09-27 11:12:59 on dist-test-slave-9hq9"
server_key: "085018c4bd61adfb1e2d3b936f65a60d"
server_key_iv: "e4d45150f22f5cc50e906022e99144a3"
server_key_version: "encryptionkey@0"
I20260927 11:12:59.261652 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:12:59.271695 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:12:59.273326 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:12:59.273470 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:12:59.273666 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:12:59.273940 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:12:59.273993 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:59.274030 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:12:59.274080 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:59.279731 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39575
I20260927 11:12:59.279804 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39575 every 8 connection(s)
I20260927 11:12:59.280086 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:12:59.281031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 1024
I20260927 11:12:59.281116 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:12:59.281275 392 external_mini_cluster.cc:1428] Setting key 227a32ee974b87d1340711b9454f8c27
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 11:12:59.289523 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.280754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34733 (local address 127.0.98.62:39183)
0927 11:12:59.280835 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:59.280838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:59.281038 (+ 200us) server_negotiation.cc:407] Connection header received
0927 11:12:59.281655 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:59.281656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:59.281667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:59.281725 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:12:59.282972 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.283643 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.284572 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.284787 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.284869 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.286849 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:12:59.286850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:12:59.286851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:12:59.286857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:12:59.288216 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.288608 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.288609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.288610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.288665 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.288914 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.288915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.288915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.289027 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:12:59.289101 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:59.289354 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 11:12:59.289437 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 11:12:59.289680 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.280725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39183 (local address 127.0.98.1:34733)
0927 11:12:59.280899 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 11:12:59.280941 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 11:12:59.281554 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:12:59.281770 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:12:59.281773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:12:59.282594 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:12:59.282873 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.282878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.283735 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.283736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.284484 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.284486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.284877 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.284878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.284999 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.285486 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:12:59.285493 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:12:59.286744 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:12:59.288319 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:12:59.288325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:12:59.288327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:12:59.288533 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:12:59.288737 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:12:59.288738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:12:59.288739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:12:59.288847 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:12:59.289093 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:12:59.289099 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:12:59.289228 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 11:12:59.289333 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 11:12:59.289470 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260927 11:12:59.290220 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39183
I20260927 11:12:59.290325 1153 heartbeater.cc:464] Registering TS with master...
I20260927 11:12:59.290485 1153 heartbeater.cc:511] Master 127.0.98.62:39183 requested a full tablet report, sending...
I20260927 11:12:59.290952 962 ts_manager.cc:194] Registered new tserver with Master: 2f3fc5db13004caeb377884dac3b34c9 (127.0.98.1:39575)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 11:12:59.291602 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:12:59.304257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39183
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:12:59.383021 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:12:59.383239 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:12:59.383278 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:12:59.384413 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:12:59.384474 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:12:59.384517 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:12:59.385860 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:12:59.386113 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:12:59.386322 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:12:59.390789 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:12:59.390807 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:12:59.390991 1158 server_base.cc:1034] running on GCE node
W20260927 11:12:59.391014 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:12:59.391237 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:12:59.391484 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:12:59.392623 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790507579392610 us; error 31 us; skew 500 ppm
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:12:59.394958 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:12:59.395390 1158 webserver.cc:460] Webserver started at http://127.0.98.2:45719/ using document root <none> and password file <none>
I20260927 11:12:59.395622 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 11:12:59.395686 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:12:59.395823 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:12:59.396770 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "cb2cf3231bd643ad89b0dda9bbfef22b"
format_stamp: "Formatted at 2026-09-27 11:12:59 on dist-test-slave-9hq9"
server_key: "33ca381261a6495a033837f0376d52ae"
server_key_iv: "5ceb3c7f354b5194bb5409f11bdcd8b0"
server_key_version: "encryptionkey@0"
I20260927 11:12:59.397104 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cb2cf3231bd643ad89b0dda9bbfef22b"
format_stamp: "Formatted at 2026-09-27 11:12:59 on dist-test-slave-9hq9"
server_key: "33ca381261a6495a033837f0376d52ae"
server_key_iv: "5ceb3c7f354b5194bb5409f11bdcd8b0"
server_key_version: "encryptionkey@0"
I20260927 11:12:59.398355 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:12:59.398998 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:59.399158 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:59.399258 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "cb2cf3231bd643ad89b0dda9bbfef22b"
format_stamp: "Formatted at 2026-09-27 11:12:59 on dist-test-slave-9hq9"
server_key: "33ca381261a6495a033837f0376d52ae"
server_key_iv: "5ceb3c7f354b5194bb5409f11bdcd8b0"
server_key_version: "encryptionkey@0"
I20260927 11:12:59.399346 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:12:59.418821 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:12:59.420954 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:12:59.421118 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:12:59.421332 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:12:59.421598 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:12:59.421651 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:59.421689 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:12:59.421717 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:59.427405 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42221
I20260927 11:12:59.427449 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42221 every 8 connection(s)
I20260927 11:12:59.427748 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:12:59.428180 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 1158
I20260927 11:12:59.428277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:12:59.428432 392 external_mini_cluster.cc:1428] Setting key 19e012384b8c637029121dda1d477884
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 11:12:59.437742 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.428426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43775 (local address 127.0.98.62:39183)
0927 11:12:59.428497 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:59.428499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:59.428687 (+ 188us) server_negotiation.cc:407] Connection header received
0927 11:12:59.429332 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:59.429333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:59.429341 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:59.429380 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:12:59.430616 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.431403 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.432326 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.432550 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.432629 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.435053 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:12:59.435054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:12:59.435055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:12:59.435060 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:12:59.436414 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.436884 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.436885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.436885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.436930 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.437185 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.437185 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.437186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.437275 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:12:59.437320 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:59.437619 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 11:12:59.437678 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 11:12:59.437934 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.428421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39183 (local address 127.0.98.2:43775)
0927 11:12:59.428562 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 11:12:59.428599 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 11:12:59.429214 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:12:59.429443 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:12:59.429446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:12:59.430261 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:12:59.430519 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.430526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.431493 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.431494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.432251 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.432253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.432646 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.432647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.432763 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.433262 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:12:59.433270 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:12:59.434964 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:12:59.436496 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:12:59.436503 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:12:59.436506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:12:59.436803 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:12:59.436992 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:12:59.436993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:12:59.436994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:12:59.437122 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:12:59.437333 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:12:59.437338 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:12:59.437487 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 11:12:59.437606 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 11:12:59.437760 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260927 11:12:59.438355 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39183
I20260927 11:12:59.438462 1287 heartbeater.cc:464] Registering TS with master...
I20260927 11:12:59.438620 1287 heartbeater.cc:511] Master 127.0.98.62:39183 requested a full tablet report, sending...
I20260927 11:12:59.438943 962 ts_manager.cc:194] Registered new tserver with Master: cb2cf3231bd643ad89b0dda9bbfef22b (127.0.98.2:42221)
I20260927 11:12:59.439304 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43775
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:12:59.452572 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39183
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:12:59.529361 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:12:59.529532 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:12:59.529553 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:12:59.530699 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:12:59.530738 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:12:59.530769 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:12:59.532028 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:12:59.532181 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:12:59.532325 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:12:59.536856 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:12:59.536867 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:12:59.536870 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:12:59.537078 1292 server_base.cc:1034] running on GCE node
I20260927 11:12:59.537231 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:12:59.537433 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:12:59.538561 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790507579538553 us; error 26 us; skew 500 ppm
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:12:59.540685 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:12:59.541067 1292 webserver.cc:460] Webserver started at http://127.0.98.3:33055/ using document root <none> and password file <none>
I20260927 11:12:59.541220 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 11:12:59.541254 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:12:59.541332 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:12:59.542255 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "1570268a8720407ab76be43209fd6520"
format_stamp: "Formatted at 2026-09-27 11:12:59 on dist-test-slave-9hq9"
server_key: "9fb6b7019fd6191ab1252ea8cbf4c687"
server_key_iv: "91b0e7c8533dd8fd934abc27f5bd6ac1"
server_key_version: "encryptionkey@0"
I20260927 11:12:59.542527 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1570268a8720407ab76be43209fd6520"
format_stamp: "Formatted at 2026-09-27 11:12:59 on dist-test-slave-9hq9"
server_key: "9fb6b7019fd6191ab1252ea8cbf4c687"
server_key_iv: "91b0e7c8533dd8fd934abc27f5bd6ac1"
server_key_version: "encryptionkey@0"
I20260927 11:12:59.543689 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:12:59.544310 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:59.544466 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:12:59.544554 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "1570268a8720407ab76be43209fd6520"
format_stamp: "Formatted at 2026-09-27 11:12:59 on dist-test-slave-9hq9"
server_key: "9fb6b7019fd6191ab1252ea8cbf4c687"
server_key_iv: "91b0e7c8533dd8fd934abc27f5bd6ac1"
server_key_version: "encryptionkey@0"
I20260927 11:12:59.544649 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:12:59.552425 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:12:59.554009 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:12:59.554195 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:12:59.554385 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:12:59.554637 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:12:59.554687 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:59.554724 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:12:59.554750 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:12:59.560189 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34705
I20260927 11:12:59.560223 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34705 every 8 connection(s)
I20260927 11:12:59.560515 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:12:59.566836 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 1292
I20260927 11:12:59.566934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:12:59.567113 392 external_mini_cluster.cc:1428] Setting key b59c9d2bb5fc33309b0f0482e1deecad
I20260927 11:12:59.568836 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.561124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59457 (local address 127.0.98.62:39183)
0927 11:12:59.561161 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:59.561163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:59.561354 (+ 191us) server_negotiation.cc:407] Connection header received
0927 11:12:59.561730 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:59.561730 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:59.561740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:59.561789 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:12:59.562631 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.563106 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.563975 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.564147 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.564197 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.566113 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:12:59.566114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:12:59.566114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:12:59.566120 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:12:59.567719 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.568119 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.568119 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.568120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.568169 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.568376 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.568376 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.568376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.568467 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:12:59.568514 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:59.568716 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 11:12:59.568777 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:12:59.568991 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.561092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39183 (local address 127.0.98.3:59457)
0927 11:12:59.561227 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 11:12:59.561263 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:12:59.561653 (+ 390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:12:59.561834 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:12:59.561837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:12:59.562395 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:12:59.562559 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.562563 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.563176 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.563177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.563911 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.563912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.564212 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.564212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.564321 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.564779 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:12:59.564786 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:12:59.565990 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:12:59.567796 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:12:59.567802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:12:59.567804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:12:59.568060 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:12:59.568236 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:12:59.568236 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:12:59.568236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:12:59.568323 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:12:59.568518 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:12:59.568521 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:12:59.568616 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 11:12:59.568691 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 11:12:59.568793 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
I20260927 11:12:59.569458 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39183
I20260927 11:12:59.569525 1421 heartbeater.cc:464] Registering TS with master...
I20260927 11:12:59.569633 1421 heartbeater.cc:511] Master 127.0.98.62:39183 requested a full tablet report, sending...
I20260927 11:12:59.569942 962 ts_manager.cc:194] Registered new tserver with Master: 1570268a8720407ab76be43209fd6520 (127.0.98.3:34705)
I20260927 11:12:59.570313 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59457
I20260927 11:12:59.578697 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:12:59.585634 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.580115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37990 (local address 127.0.98.62:39183)
0927 11:12:59.580171 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:59.580173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:59.580293 (+ 120us) server_negotiation.cc:407] Connection header received
0927 11:12:59.580364 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:59.580365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:59.580374 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:59.580437 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:12:59.581004 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.581454 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.582234 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.582418 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.582496 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.583522 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:12:59.583523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:12:59.583523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:12:59.583530 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:12:59.584542 (+ 1012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.584883 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.584884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.584885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.584944 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.585136 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.585137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.585137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.585268 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:12:59.585367 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:59.585472 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 11:12:59.585542 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:12:59.587399 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:12:59.588990 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:12:59.594878 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.590955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39575 (local address 127.0.0.1:45308)
0927 11:12:59.591233 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 11:12:59.591241 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:12:59.591272 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:12:59.591733 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:12:59.591734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:12:59.591737 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:12:59.591783 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.591784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.592947 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.592948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.593667 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.593668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.594597 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.594598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.594669 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.594680 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 11:12:59.594751 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:12:59.594797 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":34,"threads_started":1}
I20260927 11:12:59.594890 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.591220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.0.98.1:39575)
0927 11:12:59.591552 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:59.591556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:59.591565 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:12:59.591604 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:59.591606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:59.591620 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:59.591849 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:12:59.591914 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.592852 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.593763 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.594508 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.594580 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.594630 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:59.594774 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 11:12:59.594810 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":91,"threads_started":1}
I20260927 11:12:59.595108 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.590924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34705 (local address 127.0.0.1:44764)
0927 11:12:59.591052 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 11:12:59.591084 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 11:12:59.591149 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:12:59.591672 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:12:59.591673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:12:59.591675 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:12:59.591711 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.591713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.592667 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.592668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.593955 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.593957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.594740 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.594741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.594793 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.594795 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:12:59.594843 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 11:12:59.594874 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":84,"thread_start_us":41,"threads_started":1}
I20260927 11:12:59.595242 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.591391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44764 (local address 127.0.98.3:34705)
0927 11:12:59.591537 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:59.591547 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:59.591555 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:12:59.591580 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:59.591581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:59.591592 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:59.591674 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:12:59.591764 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.592552 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.594042 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.594586 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.594649 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.594699 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:59.594915 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 11:12:59.594955 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260927 11:12:59.595402 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.590925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42221 (local address 127.0.0.1:58496)
0927 11:12:59.591117 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 11:12:59.591125 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:12:59.591165 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:12:59.591355 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:12:59.591357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:12:59.591359 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:12:59.591409 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.591410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.592637 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.592639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.594093 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.594095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.595093 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.595094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.595248 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.595252 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:12:59.595298 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 11:12:59.595333 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":38,"thread_start_us":49,"threads_started":1}
I20260927 11:12:59.595542 1087 tablet_service.cc:1467] Processing CreateTablet for tablet bcb68b73b5514c769d0848ba4ac40c13 (DEFAULT_TABLE table=test-table [id=1304b0dea1b54fa583e6fe8aaa968591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:12:59.595716 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.591035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58496 (local address 127.0.98.2:42221)
0927 11:12:59.591172 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:59.591176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:59.591187 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:12:59.591229 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:59.591232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:59.591247 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:59.591336 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:12:59.591463 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.592537 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.594167 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.594866 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.595483 (+ 617us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.595541 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:59.595572 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 11:12:59.595638 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":73,"threads_started":1}
I20260927 11:12:59.595880 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb68b73b5514c769d0848ba4ac40c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:12:59.598325 1435 tablet_bootstrap.cc:492] T bcb68b73b5514c769d0848ba4ac40c13 P 2f3fc5db13004caeb377884dac3b34c9: Bootstrap starting.
I20260927 11:12:59.598608 1221 tablet_service.cc:1467] Processing CreateTablet for tablet bcb68b73b5514c769d0848ba4ac40c13 (DEFAULT_TABLE table=test-table [id=1304b0dea1b54fa583e6fe8aaa968591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:12:59.598850 1435 tablet_bootstrap.cc:654] T bcb68b73b5514c769d0848ba4ac40c13 P 2f3fc5db13004caeb377884dac3b34c9: Neither blocks nor log segments found. Creating new log.
I20260927 11:12:59.598836 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb68b73b5514c769d0848ba4ac40c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:12:59.599370 1435 log.cc:824] T bcb68b73b5514c769d0848ba4ac40c13 P 2f3fc5db13004caeb377884dac3b34c9: Log is configured to *not* fsync() on all Append() calls
I20260927 11:12:59.599985 1435 tablet_bootstrap.cc:492] T bcb68b73b5514c769d0848ba4ac40c13 P 2f3fc5db13004caeb377884dac3b34c9: No bootstrap required, opened a new log
I20260927 11:12:59.600024 1435 ts_tablet_manager.cc:1397] T bcb68b73b5514c769d0848ba4ac40c13 P 2f3fc5db13004caeb377884dac3b34c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:12:59.601194 1437 tablet_bootstrap.cc:492] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b: Bootstrap starting.
I20260927 11:12:59.601219 1435 raft_consensus.cc:348] T bcb68b73b5514c769d0848ba4ac40c13 P 2f3fc5db13004caeb377884dac3b34c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2cf3231bd643ad89b0dda9bbfef22b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } } peers { permanent_uuid: "1570268a8720407ab76be43209fd6520" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34705 } } peers { permanent_uuid: "2f3fc5db13004caeb377884dac3b34c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39575 } }
I20260927 11:12:59.601389 1435 raft_consensus.cc:374] T bcb68b73b5514c769d0848ba4ac40c13 P 2f3fc5db13004caeb377884dac3b34c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:12:59.601423 1435 raft_consensus.cc:729] T bcb68b73b5514c769d0848ba4ac40c13 P 2f3fc5db13004caeb377884dac3b34c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3fc5db13004caeb377884dac3b34c9, State: Initialized, Role: FOLLOWER
I20260927 11:12:59.601531 1435 consensus_queue.cc:260] T bcb68b73b5514c769d0848ba4ac40c13 P 2f3fc5db13004caeb377884dac3b34c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2cf3231bd643ad89b0dda9bbfef22b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } } peers { permanent_uuid: "1570268a8720407ab76be43209fd6520" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34705 } } peers { permanent_uuid: "2f3fc5db13004caeb377884dac3b34c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39575 } }
I20260927 11:12:59.601564 1355 tablet_service.cc:1467] Processing CreateTablet for tablet bcb68b73b5514c769d0848ba4ac40c13 (DEFAULT_TABLE table=test-table [id=1304b0dea1b54fa583e6fe8aaa968591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:12:59.601810 1437 tablet_bootstrap.cc:654] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b: Neither blocks nor log segments found. Creating new log.
I20260927 11:12:59.601840 1153 heartbeater.cc:503] Master 127.0.98.62:39183 was elected leader, sending a full tablet report...
I20260927 11:12:59.601853 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb68b73b5514c769d0848ba4ac40c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:12:59.602025 1435 ts_tablet_manager.cc:1428] T bcb68b73b5514c769d0848ba4ac40c13 P 2f3fc5db13004caeb377884dac3b34c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:12:59.602371 1437 log.cc:824] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b: Log is configured to *not* fsync() on all Append() calls
I20260927 11:12:59.604821 1440 tablet_bootstrap.cc:492] T bcb68b73b5514c769d0848ba4ac40c13 P 1570268a8720407ab76be43209fd6520: Bootstrap starting.
I20260927 11:12:59.605374 1437 tablet_bootstrap.cc:492] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b: No bootstrap required, opened a new log
I20260927 11:12:59.605388 1440 tablet_bootstrap.cc:654] T bcb68b73b5514c769d0848ba4ac40c13 P 1570268a8720407ab76be43209fd6520: Neither blocks nor log segments found. Creating new log.
I20260927 11:12:59.605455 1437 ts_tablet_manager.cc:1397] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 11:12:59.605794 1440 log.cc:824] T bcb68b73b5514c769d0848ba4ac40c13 P 1570268a8720407ab76be43209fd6520: Log is configured to *not* fsync() on all Append() calls
I20260927 11:12:59.606449 1440 tablet_bootstrap.cc:492] T bcb68b73b5514c769d0848ba4ac40c13 P 1570268a8720407ab76be43209fd6520: No bootstrap required, opened a new log
I20260927 11:12:59.606498 1440 ts_tablet_manager.cc:1397] T bcb68b73b5514c769d0848ba4ac40c13 P 1570268a8720407ab76be43209fd6520: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:12:59.607203 1437 raft_consensus.cc:348] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2cf3231bd643ad89b0dda9bbfef22b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } } peers { permanent_uuid: "1570268a8720407ab76be43209fd6520" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34705 } } peers { permanent_uuid: "2f3fc5db13004caeb377884dac3b34c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39575 } }
I20260927 11:12:59.607344 1437 raft_consensus.cc:374] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:12:59.607399 1437 raft_consensus.cc:729] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb2cf3231bd643ad89b0dda9bbfef22b, State: Initialized, Role: FOLLOWER
I20260927 11:12:59.607498 1437 consensus_queue.cc:260] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2cf3231bd643ad89b0dda9bbfef22b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } } peers { permanent_uuid: "1570268a8720407ab76be43209fd6520" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34705 } } peers { permanent_uuid: "2f3fc5db13004caeb377884dac3b34c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39575 } }
I20260927 11:12:59.607790 1437 ts_tablet_manager.cc:1428] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:12:59.607803 1287 heartbeater.cc:503] Master 127.0.98.62:39183 was elected leader, sending a full tablet report...
I20260927 11:12:59.608322 1440 raft_consensus.cc:348] T bcb68b73b5514c769d0848ba4ac40c13 P 1570268a8720407ab76be43209fd6520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2cf3231bd643ad89b0dda9bbfef22b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } } peers { permanent_uuid: "1570268a8720407ab76be43209fd6520" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34705 } } peers { permanent_uuid: "2f3fc5db13004caeb377884dac3b34c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39575 } }
I20260927 11:12:59.608500 1440 raft_consensus.cc:374] T bcb68b73b5514c769d0848ba4ac40c13 P 1570268a8720407ab76be43209fd6520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:12:59.608546 1440 raft_consensus.cc:729] T bcb68b73b5514c769d0848ba4ac40c13 P 1570268a8720407ab76be43209fd6520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1570268a8720407ab76be43209fd6520, State: Initialized, Role: FOLLOWER
I20260927 11:12:59.608655 1440 consensus_queue.cc:260] T bcb68b73b5514c769d0848ba4ac40c13 P 1570268a8720407ab76be43209fd6520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2cf3231bd643ad89b0dda9bbfef22b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } } peers { permanent_uuid: "1570268a8720407ab76be43209fd6520" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34705 } } peers { permanent_uuid: "2f3fc5db13004caeb377884dac3b34c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39575 } }
I20260927 11:12:59.608906 1440 ts_tablet_manager.cc:1428] T bcb68b73b5514c769d0848ba4ac40c13 P 1570268a8720407ab76be43209fd6520: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:12:59.608932 1421 heartbeater.cc:503] Master 127.0.98.62:39183 was elected leader, sending a full tablet report...
I20260927 11:12:59.643105 1442 raft_consensus.cc:482] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:12:59.643179 1442 raft_consensus.cc:504] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2cf3231bd643ad89b0dda9bbfef22b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } } peers { permanent_uuid: "1570268a8720407ab76be43209fd6520" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34705 } } peers { permanent_uuid: "2f3fc5db13004caeb377884dac3b34c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39575 } }
I20260927 11:12:59.643503 1442 leader_election.cc:288] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1570268a8720407ab76be43209fd6520 (127.0.98.3:34705), 2f3fc5db13004caeb377884dac3b34c9 (127.0.98.1:39575)
I20260927 11:12:59.647171 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.643732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49191 (local address 127.0.98.3:34705)
0927 11:12:59.643765 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:59.643767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:59.643776 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:12:59.643797 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:59.643797 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:59.643813 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:59.643890 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:12:59.644058 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.644772 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.646262 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.646877 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.646936 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.646946 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:59.647080 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 11:12:59.647112 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:12:59.647171 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.643695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39963 (local address 127.0.98.1:39575)
0927 11:12:59.643747 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:59.643748 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:59.643844 (+ 96us) server_negotiation.cc:407] Connection header received
0927 11:12:59.643877 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:59.643878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:59.643888 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:59.643937 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:12:59.644062 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.644768 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.646263 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.646826 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.646897 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.646908 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:59.647080 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 11:12:59.647112 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:12:59.647202 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.643625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34705 (local address 127.0.98.2:49191)
0927 11:12:59.643672 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 11:12:59.643685 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:12:59.643727 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:12:59.643902 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:12:59.643903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:12:59.643904 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:12:59.643967 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.643968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.644846 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.644847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.646183 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.646186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.646943 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.646944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.647011 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.647013 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:12:59.647070 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 11:12:59.647115 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":74}
I20260927 11:12:59.647209 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.643651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39575 (local address 127.0.98.2:39963)
0927 11:12:59.643777 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 11:12:59.643785 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:12:59.643837 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:12:59.643946 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:12:59.643947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:12:59.643948 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:12:59.643998 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.643999 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.644846 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.644847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.646184 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:12:59.646186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.646900 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.646901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:12:59.647009 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.647014 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:12:59.647070 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 11:12:59.647114 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":41,"thread_start_us":69,"threads_started":1}
I20260927 11:12:59.647523 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb68b73b5514c769d0848ba4ac40c13" candidate_uuid: "cb2cf3231bd643ad89b0dda9bbfef22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1570268a8720407ab76be43209fd6520" is_pre_election: true
I20260927 11:12:59.647552 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb68b73b5514c769d0848ba4ac40c13" candidate_uuid: "cb2cf3231bd643ad89b0dda9bbfef22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3fc5db13004caeb377884dac3b34c9" is_pre_election: true
I20260927 11:12:59.647661 1375 raft_consensus.cc:2454] T bcb68b73b5514c769d0848ba4ac40c13 P 1570268a8720407ab76be43209fd6520 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb2cf3231bd643ad89b0dda9bbfef22b in term 0.
I20260927 11:12:59.647681 1107 raft_consensus.cc:2454] T bcb68b73b5514c769d0848ba4ac40c13 P 2f3fc5db13004caeb377884dac3b34c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb2cf3231bd643ad89b0dda9bbfef22b in term 0.
I20260927 11:12:59.647852 1177 leader_election.cc:302] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1570268a8720407ab76be43209fd6520, cb2cf3231bd643ad89b0dda9bbfef22b; no voters:
I20260927 11:12:59.647985 1442 raft_consensus.cc:2781] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:12:59.648061 1442 raft_consensus.cc:482] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:12:59.648090 1442 raft_consensus.cc:3037] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:12:59.648767 1442 raft_consensus.cc:504] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2cf3231bd643ad89b0dda9bbfef22b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } } peers { permanent_uuid: "1570268a8720407ab76be43209fd6520" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34705 } } peers { permanent_uuid: "2f3fc5db13004caeb377884dac3b34c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39575 } }
I20260927 11:12:59.648892 1442 leader_election.cc:288] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [CANDIDATE]: Term 1 election: Requested vote from peers 1570268a8720407ab76be43209fd6520 (127.0.98.3:34705), 2f3fc5db13004caeb377884dac3b34c9 (127.0.98.1:39575)
I20260927 11:12:59.648998 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb68b73b5514c769d0848ba4ac40c13" candidate_uuid: "cb2cf3231bd643ad89b0dda9bbfef22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1570268a8720407ab76be43209fd6520"
I20260927 11:12:59.649013 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb68b73b5514c769d0848ba4ac40c13" candidate_uuid: "cb2cf3231bd643ad89b0dda9bbfef22b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3fc5db13004caeb377884dac3b34c9"
I20260927 11:12:59.649060 1375 raft_consensus.cc:3037] T bcb68b73b5514c769d0848ba4ac40c13 P 1570268a8720407ab76be43209fd6520 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:12:59.649092 1107 raft_consensus.cc:3037] T bcb68b73b5514c769d0848ba4ac40c13 P 2f3fc5db13004caeb377884dac3b34c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:12:59.649734 1375 raft_consensus.cc:2454] T bcb68b73b5514c769d0848ba4ac40c13 P 1570268a8720407ab76be43209fd6520 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb2cf3231bd643ad89b0dda9bbfef22b in term 1.
I20260927 11:12:59.649732 1107 raft_consensus.cc:2454] T bcb68b73b5514c769d0848ba4ac40c13 P 2f3fc5db13004caeb377884dac3b34c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb2cf3231bd643ad89b0dda9bbfef22b in term 1.
I20260927 11:12:59.649899 1175 leader_election.cc:302] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3fc5db13004caeb377884dac3b34c9, cb2cf3231bd643ad89b0dda9bbfef22b; no voters:
I20260927 11:12:59.649994 1442 raft_consensus.cc:2781] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:12:59.650146 1442 raft_consensus.cc:686] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [term 1 LEADER]: Becoming Leader. State: Replica: cb2cf3231bd643ad89b0dda9bbfef22b, State: Running, Role: LEADER
I20260927 11:12:59.650208 1442 consensus_queue.cc:237] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2cf3231bd643ad89b0dda9bbfef22b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } } peers { permanent_uuid: "1570268a8720407ab76be43209fd6520" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34705 } } peers { permanent_uuid: "2f3fc5db13004caeb377884dac3b34c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39575 } }
I20260927 11:12:59.650900 962 catalog_manager.cc:5495] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b reported cstate change: term changed from 0 to 1, leader changed from <none> to cb2cf3231bd643ad89b0dda9bbfef22b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "cb2cf3231bd643ad89b0dda9bbfef22b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb2cf3231bd643ad89b0dda9bbfef22b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1570268a8720407ab76be43209fd6520" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f3fc5db13004caeb377884dac3b34c9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39575 } health_report { overall_health: UNKNOWN } } }
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:12:59.670148 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.663846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45312 (local address 127.0.98.1:39575)
0927 11:12:59.663899 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:59.663901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:59.663971 (+ 70us) server_negotiation.cc:407] Connection header received
0927 11:12:59.664049 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:59.664049 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:59.664057 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:59.664115 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:12:59.664617 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.665049 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.665646 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.665840 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.665883 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.667990 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:12:59.667996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:12:59.667997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:12:59.668002 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:12:59.669283 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.669562 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.669565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.669565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.669611 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.669747 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.669747 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.669748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.669849 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:12:59.669937 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:59.669991 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 11:12:59.670044 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 11:12:59.678220 1288 tablet.cc:2369] T bcb68b73b5514c769d0848ba4ac40c13 P cb2cf3231bd643ad89b0dda9bbfef22b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507579, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507579, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:12:59.691268 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.684449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.0.98.1:39575)
0927 11:12:59.684506 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:59.684508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:59.684598 (+ 90us) server_negotiation.cc:407] Connection header received
0927 11:12:59.684686 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:59.684687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:59.684711 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:59.684764 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:12:59.685288 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.685732 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.686500 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.686683 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.686766 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.688783 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:12:59.688784 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:12:59.688785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:12:59.688791 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:12:59.690171 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.690512 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.690513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.690513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.690566 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.690758 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.690759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.690760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.690848 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:12:59.690949 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:59.691105 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 11:12:59.691176 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:12:59.712854 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:12:59.704251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45332 (local address 127.0.98.1:39575)
0927 11:12:59.704329 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 11:12:59.704331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:12:59.704360 (+ 29us) server_negotiation.cc:407] Connection header received
0927 11:12:59.704418 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:12:59.704418 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:12:59.704425 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:12:59.704469 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:12:59.705050 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.705776 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.706924 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:12:59.707251 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:12:59.707311 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:12:59.710019 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:12:59.710020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:12:59.710021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:12:59.710028 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:12:59.711637 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.711960 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.711961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.711961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.712012 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:12:59.712274 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:12:59.712275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:12:59.712275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:12:59.712442 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:12:59.712534 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 11:12:59.712743 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 11:12:59.712798 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 11:12:59.713382 1067 tablet_service.cc:2908] Rejecting scan request for tablet bcb68b73b5514c769d0848ba4ac40c13: Uninitialized: clean time has not yet been initialized
I20260927 11:12:59.714150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 1024
I20260927 11:12:59.719349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 1158
I20260927 11:12:59.723511 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 1292
I20260927 11:12:59.727553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 930
2026-09-27T11:12:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1887 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[1459](info): set up 2 sockets
Sep 27 11:12:59 dist-test-slave-9hq9 krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260927 11:13:00.741405 392 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:13:00 dist-test-slave-9hq9 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:13:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:13:00.845862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36307
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:00.921574 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:00.921778 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:00.921813 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:00.922984 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:00.923046 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 11:13:00.923071 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:00.923103 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:00.923130 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:00.923163 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:00.924588 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36307
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:00.924844 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:00.925048 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:00.929586 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:00.929610 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:00.929648 1472 server_base.cc:1034] running on GCE node
W20260927 11:13:00.929617 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:00.929960 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:00.930277 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:00.931427 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790507580931416 us; error 37 us; skew 500 ppm
Sep 27 11:13:00 dist-test-slave-9hq9 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:00.933734 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:13:00.934181 1472 webserver.cc:460] Webserver started at http://127.0.98.62:40817/ using document root <none> and password file <none>
I20260927 11:13:00.934379 1472 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:00.934441 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:00.934580 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:00.935477 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "61458cfbbaae4051803d1dfa628b6c89"
format_stamp: "Formatted at 2026-09-27 11:13:00 on dist-test-slave-9hq9"
server_key: "312beac837ec0d404f3147e7e420f55d"
server_key_iv: "88dbc33ad8319828a744159a9883fba5"
server_key_version: "encryptionkey@0"
I20260927 11:13:00.935837 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "61458cfbbaae4051803d1dfa628b6c89"
format_stamp: "Formatted at 2026-09-27 11:13:00 on dist-test-slave-9hq9"
server_key: "312beac837ec0d404f3147e7e420f55d"
server_key_iv: "88dbc33ad8319828a744159a9883fba5"
server_key_version: "encryptionkey@0"
I20260927 11:13:00.937057 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:13:00.937677 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:00.937835 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:00.937925 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "61458cfbbaae4051803d1dfa628b6c89"
format_stamp: "Formatted at 2026-09-27 11:13:00 on dist-test-slave-9hq9"
server_key: "312beac837ec0d404f3147e7e420f55d"
server_key_iv: "88dbc33ad8319828a744159a9883fba5"
server_key_version: "encryptionkey@0"
I20260927 11:13:00.938006 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:00.955679 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:00.957798 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:00.957939 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:00.961992 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36307
I20260927 11:13:00.962105 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36307 every 8 connection(s)
I20260927 11:13:00.962369 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:00.962946 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:00.964931 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89: Bootstrap starting.
I20260927 11:13:00.965489 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:00.965781 1550 log.cc:824] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:00.966442 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89: No bootstrap required, opened a new log
I20260927 11:13:00.968358 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61458cfbbaae4051803d1dfa628b6c89" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } }
I20260927 11:13:00.968477 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:00.968521 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61458cfbbaae4051803d1dfa628b6c89, State: Initialized, Role: FOLLOWER
I20260927 11:13:00.968665 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61458cfbbaae4051803d1dfa628b6c89" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } }
I20260927 11:13:00.968724 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:00.968743 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:00.968809 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:00.969302 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61458cfbbaae4051803d1dfa628b6c89" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } }
I20260927 11:13:00.969378 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61458cfbbaae4051803d1dfa628b6c89; no voters:
I20260927 11:13:00.969518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 1472
I20260927 11:13:00.969527 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:00.969559 1553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:00.969626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:00.969666 1553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [term 1 LEADER]: Becoming Leader. State: Replica: 61458cfbbaae4051803d1dfa628b6c89, State: Running, Role: LEADER
I20260927 11:13:00.969748 1553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61458cfbbaae4051803d1dfa628b6c89" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } }
I20260927 11:13:00.969784 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:00.969792 392 external_mini_cluster.cc:1428] Setting key 1b01c0e21dc6276a651b6dcdce0adf77
I20260927 11:13:00.970141 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61458cfbbaae4051803d1dfa628b6c89. Latest consensus state: current_term: 1 leader_uuid: "61458cfbbaae4051803d1dfa628b6c89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61458cfbbaae4051803d1dfa628b6c89" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } } }
I20260927 11:13:00.970221 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:00.970377 1554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61458cfbbaae4051803d1dfa628b6c89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61458cfbbaae4051803d1dfa628b6c89" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36307 } } }
I20260927 11:13:00.970481 1554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:00.970949 1559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:00.971428 1559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:00.972975 1559 catalog_manager.cc:1347] Generated new cluster ID: 41b3a4eb600a44ee9acc876c6334f60b
I20260927 11:13:00.973026 1559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:13:00 dist-test-slave-9hq9 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:00.979995 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:00.970175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.0.98.62:36307)
0927 11:13:00.970473 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:00.970482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:00.970516 (+ 34us) server_negotiation.cc:407] Connection header received
0927 11:13:00.970634 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:00.970640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:00.970702 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:00.971604 (+ 902us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:00.971706 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:00.972842 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:00.973661 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:00.974040 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:00.974451 (+ 411us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:00.976415 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:00.976419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:00.976420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:00.976431 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:00.978510 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:00.978890 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:00.978892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:00.978893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:00.978973 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:00.979188 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:00.979189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:00.979189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:00.979402 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:00.979565 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:00.979627 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 11:13:00.979860 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":68,"threads_started":1}
I20260927 11:13:00.988463 1559 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:00.988909 1559 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:00.997340 1559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89: Generated new TSK 0
I20260927 11:13:00.997500 1559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:13:01.025177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36307
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:01.124899 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:01.125090 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:01.125113 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:01.126353 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:01.126389 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:01.126410 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:13:01.126426 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:01.127918 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:01.128119 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:01.128328 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:01.132880 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:01.132910 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:01.132903 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:01.133095 1576 server_base.cc:1034] running on GCE node
I20260927 11:13:01.133361 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:01.133608 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:01.134761 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790507581134756 us; error 35 us; skew 500 ppm
Sep 27 11:13:01 dist-test-slave-9hq9 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:01.136906 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:13:01.137283 1576 webserver.cc:460] Webserver started at http://127.0.98.1:34047/ using document root <none> and password file <none>
I20260927 11:13:01.137497 1576 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:01.137562 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:01.137709 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:01.138723 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "6f7eb1a45adf40ab91828d6a6aad69d9"
format_stamp: "Formatted at 2026-09-27 11:13:01 on dist-test-slave-9hq9"
server_key: "3c21625100d1daecfd09fbed2e3c8982"
server_key_iv: "538b6b31de094ee5338602872c7e2c26"
server_key_version: "encryptionkey@0"
I20260927 11:13:01.139058 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6f7eb1a45adf40ab91828d6a6aad69d9"
format_stamp: "Formatted at 2026-09-27 11:13:01 on dist-test-slave-9hq9"
server_key: "3c21625100d1daecfd09fbed2e3c8982"
server_key_iv: "538b6b31de094ee5338602872c7e2c26"
server_key_version: "encryptionkey@0"
I20260927 11:13:01.140326 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:13:01.141028 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:01.141184 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:01.141312 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "6f7eb1a45adf40ab91828d6a6aad69d9"
format_stamp: "Formatted at 2026-09-27 11:13:01 on dist-test-slave-9hq9"
server_key: "3c21625100d1daecfd09fbed2e3c8982"
server_key_iv: "538b6b31de094ee5338602872c7e2c26"
server_key_version: "encryptionkey@0"
I20260927 11:13:01.141383 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:01.152704 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:01.154376 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:01.154515 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:01.154968 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:01.155040 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:01.155090 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:01.155134 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:01.161286 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44873
I20260927 11:13:01.161392 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44873 every 8 connection(s)
I20260927 11:13:01.161640 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:13:01 dist-test-slave-9hq9 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:01.164424 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.155484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56861 (local address 127.0.98.62:36307)
0927 11:13:01.155526 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:01.155528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:01.155732 (+ 204us) server_negotiation.cc:407] Connection header received
0927 11:13:01.156176 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:01.156177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:01.156188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:01.156248 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:01.157030 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.157836 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.159395 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.159733 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.159843 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.162299 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:01.162301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:01.162301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:01.162308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:01.163431 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:01.163768 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:01.163768 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:01.163769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:01.163816 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:01.163976 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:01.163976 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:01.163976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:01.164062 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:01.164115 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:01.164312 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 11:13:01.164367 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:13:01.164610 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.155476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36307 (local address 127.0.98.1:56861)
0927 11:13:01.155627 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:01.155655 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:01.156080 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:01.156310 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:01.156314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:01.156658 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:01.156912 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.156919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.157928 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.157929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.159279 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.159282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.159842 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.159843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.159935 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.160409 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:01.160417 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:01.162193 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:01.163495 (+ 1302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:01.163501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:01.163502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:01.163703 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:01.163856 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:01.163856 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:01.163856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:01.163929 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:01.164123 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:01.164128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:01.164223 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 11:13:01.164294 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:13:01.164394 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260927 11:13:01.165019 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36307
I20260927 11:13:01.165092 1707 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:01.165266 1707 heartbeater.cc:511] Master 127.0.98.62:36307 requested a full tablet report, sending...
I20260927 11:13:01.165792 1503 ts_manager.cc:194] Registered new tserver with Master: 6f7eb1a45adf40ab91828d6a6aad69d9 (127.0.98.1:44873)
I20260927 11:13:01.166664 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56861
I20260927 11:13:01.169646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 1576
I20260927 11:13:01.169737 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:13:01.169904 392 external_mini_cluster.cc:1428] Setting key 160b487b2afbf0c6d723d1c70416a3a8
I20260927 11:13:01.171406 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.166122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40021 (local address 127.0.98.62:36307)
0927 11:13:01.166153 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:01.166155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:01.166163 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:13:01.166181 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:01.166182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:01.166190 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:01.166255 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:01.166502 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.167124 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.167853 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.168020 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.168073 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.168810 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:01.168811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:01.168811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:01.168817 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:01.170357 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:01.170750 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:01.170751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:01.170751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:01.170805 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:01.171008 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:01.171008 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:01.171009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:01.171092 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:01.171134 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:01.171298 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 11:13:01.171354 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:13:01.171527 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.165914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36307 (local address 127.0.98.1:40021)
0927 11:13:01.166019 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:01.166029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:01.166093 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:01.166257 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:01.166258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:01.166392 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:01.166441 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.166442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.167213 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.167214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.167785 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.167787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.168077 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.168078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.168126 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.168463 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:01.168467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:01.168741 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:01.170433 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:01.170434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:01.170434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:01.170684 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:01.170880 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:01.170881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:01.170881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:01.170948 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:01.171154 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:01.171155 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:01.171202 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 11:13:01.171273 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:13:01.171376 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:13:01.192350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36307
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:01.268011 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:01.268210 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:01.268249 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:01.269415 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:01.269465 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:01.269496 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:13:01.269541 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:01.271087 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:01.271325 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:01.271559 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:13:01.276196 1716 server_base.cc:1034] running on GCE node
W20260927 11:13:01.276156 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:01.276134 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:01.276146 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:01.276489 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:01.276700 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:01.277858 1716 hybrid_clock.cc:648] HybridClock initialized: now 1790507581277847 us; error 25 us; skew 500 ppm
Sep 27 11:13:01 dist-test-slave-9hq9 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:01.280112 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:13:01.280480 1716 webserver.cc:460] Webserver started at http://127.0.98.2:43621/ using document root <none> and password file <none>
I20260927 11:13:01.280644 1716 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:01.280678 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:01.280750 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:01.281617 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "db82beb4a4344b898bcf19f4e020cd9d"
format_stamp: "Formatted at 2026-09-27 11:13:01 on dist-test-slave-9hq9"
server_key: "131688dfaefae67704239da8f558069c"
server_key_iv: "be0dacce4e67d50cd9472471b2559157"
server_key_version: "encryptionkey@0"
I20260927 11:13:01.281877 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "db82beb4a4344b898bcf19f4e020cd9d"
format_stamp: "Formatted at 2026-09-27 11:13:01 on dist-test-slave-9hq9"
server_key: "131688dfaefae67704239da8f558069c"
server_key_iv: "be0dacce4e67d50cd9472471b2559157"
server_key_version: "encryptionkey@0"
I20260927 11:13:01.283097 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:13:01.283716 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:01.283870 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:01.283962 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "db82beb4a4344b898bcf19f4e020cd9d"
format_stamp: "Formatted at 2026-09-27 11:13:01 on dist-test-slave-9hq9"
server_key: "131688dfaefae67704239da8f558069c"
server_key_iv: "be0dacce4e67d50cd9472471b2559157"
server_key_version: "encryptionkey@0"
I20260927 11:13:01.284035 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:01.316911 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:01.319072 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:01.319238 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:01.319766 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:01.319810 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:01.319839 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:01.319855 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 11:13:01 dist-test-slave-9hq9 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:01.326756 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34377
I20260927 11:13:01.327047 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:13:01.327071 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34377 every 8 connection(s)
I20260927 11:13:01.328980 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.320086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53099 (local address 127.0.98.62:36307)
0927 11:13:01.320154 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:01.320156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:01.320369 (+ 213us) server_negotiation.cc:407] Connection header received
0927 11:13:01.320971 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:01.320973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:01.320982 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:01.321022 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:01.321827 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.322458 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.323460 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.323660 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.323733 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.326525 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:01.326527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:01.326527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:01.326534 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:01.328001 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:01.328337 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:01.328338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:01.328338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:01.328386 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:01.328544 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:01.328544 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:01.328544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:01.328645 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:01.328689 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:01.328874 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 11:13:01.328925 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 11:13:01.329123 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.320083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36307 (local address 127.0.98.2:53099)
0927 11:13:01.320230 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:01.320273 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:01.320855 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:01.321090 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:01.321094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:01.321459 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:01.321729 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.321736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.322558 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.322559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.323361 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.323363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.323752 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.323753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.323877 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.324358 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:01.324373 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:01.326425 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:01.328077 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:01.328080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:01.328082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:01.328274 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:01.328425 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:01.328425 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:01.328425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:01.328500 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:01.328694 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:01.328697 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:01.328791 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 11:13:01.328862 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:13:01.328960 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260927 11:13:01.329521 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36307
I20260927 11:13:01.329603 1847 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:01.329792 1847 heartbeater.cc:511] Master 127.0.98.62:36307 requested a full tablet report, sending...
I20260927 11:13:01.330163 1504 ts_manager.cc:194] Registered new tserver with Master: db82beb4a4344b898bcf19f4e020cd9d (127.0.98.2:34377)
I20260927 11:13:01.330570 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53099
I20260927 11:13:01.335160 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.330567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34835 (local address 127.0.98.62:36307)
0927 11:13:01.330613 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:01.330614 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:01.330730 (+ 116us) server_negotiation.cc:407] Connection header received
0927 11:13:01.330750 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:01.330751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:01.330760 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:01.330816 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:01.331161 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.331592 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.332220 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.332378 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.332421 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.333220 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:01.333220 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:01.333221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:01.333225 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:01.334325 (+ 1100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:01.334601 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:01.334601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:01.334601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:01.334672 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:01.334810 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:01.334810 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:01.334810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:01.334917 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:01.334965 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:01.335055 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 11:13:01.335109 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:13:01.335168 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.330488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36307 (local address 127.0.98.2:34835)
0927 11:13:01.330610 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:01.330621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:01.330681 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:01.330819 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:01.330820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:01.331039 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:01.331093 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.331095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.331672 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.331673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.332158 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.332159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.332450 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.332451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.332494 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.332899 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:01.332904 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:01.333156 (+ 252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:01.334386 (+ 1230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:01.334387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:01.334387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:01.334531 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:01.334728 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:01.334728 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:01.334728 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:01.334773 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:01.334965 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:01.334965 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:01.334994 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 11:13:01.335042 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 11:13:01.335107 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260927 11:13:01.336629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 1716
I20260927 11:13:01.336710 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:13:01.336884 392 external_mini_cluster.cc:1428] Setting key 393ca2f584d0cc5d2e09b782df722cb6
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:13:01.358263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36307
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:01.435324 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:01.435500 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:01.435541 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:01.436650 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:01.436689 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:01.436709 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:13:01.436758 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:01.438201 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:01.438402 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:01.438612 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:01.443328 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:01.443346 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:01.443363 1856 server_base.cc:1034] running on GCE node
W20260927 11:13:01.443347 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:01.443717 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:01.443948 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:01.445101 1856 hybrid_clock.cc:648] HybridClock initialized: now 1790507581445079 us; error 36 us; skew 500 ppm
Sep 27 11:13:01 dist-test-slave-9hq9 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:01.447489 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:13:01.447873 1856 webserver.cc:460] Webserver started at http://127.0.98.3:44047/ using document root <none> and password file <none>
I20260927 11:13:01.448072 1856 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:01.448117 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:01.448251 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:01.449188 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "ff39cfe91c9c4452989c52e52a1ad592"
format_stamp: "Formatted at 2026-09-27 11:13:01 on dist-test-slave-9hq9"
server_key: "9ef762e73e32fa74eae44efab1f2981f"
server_key_iv: "f93fdc5e006202e820357117e6f40505"
server_key_version: "encryptionkey@0"
I20260927 11:13:01.449570 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ff39cfe91c9c4452989c52e52a1ad592"
format_stamp: "Formatted at 2026-09-27 11:13:01 on dist-test-slave-9hq9"
server_key: "9ef762e73e32fa74eae44efab1f2981f"
server_key_iv: "f93fdc5e006202e820357117e6f40505"
server_key_version: "encryptionkey@0"
I20260927 11:13:01.450932 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:13:01.451609 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:01.451790 1856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:01.451881 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "ff39cfe91c9c4452989c52e52a1ad592"
format_stamp: "Formatted at 2026-09-27 11:13:01 on dist-test-slave-9hq9"
server_key: "9ef762e73e32fa74eae44efab1f2981f"
server_key_iv: "f93fdc5e006202e820357117e6f40505"
server_key_version: "encryptionkey@0"
I20260927 11:13:01.451943 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:01.459766 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:01.461856 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:01.461990 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:01.462481 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:01.462519 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:01.462541 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:01.462585 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:01.468830 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35695
Sep 27 11:13:01 dist-test-slave-9hq9 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:01.469163 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:13:01.469381 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35695 every 8 connection(s)
I20260927 11:13:01.472205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 1856
I20260927 11:13:01.472291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:13:01.472455 392 external_mini_cluster.cc:1428] Setting key b4dd48cd1418d05ec0ce64d09bd8b235
I20260927 11:13:01.472591 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.462946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52895 (local address 127.0.98.62:36307)
0927 11:13:01.463021 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:01.463024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:01.463172 (+ 148us) server_negotiation.cc:407] Connection header received
0927 11:13:01.463585 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:01.463586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:01.463600 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:01.463677 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:01.464454 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.465298 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.466892 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.467240 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.467311 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.469755 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:01.469756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:01.469756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:01.469763 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:01.471305 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:01.471672 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:01.471673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:01.471673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:01.471732 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:01.471938 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:01.471938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:01.471939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:01.472035 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:01.472100 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:01.472404 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 11:13:01.472463 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:13:01.472764 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.462905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36307 (local address 127.0.98.3:52895)
0927 11:13:01.463049 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:01.463085 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:01.463498 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:01.463736 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:01.463738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:01.464082 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:01.464353 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.464359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.465396 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.465397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.466785 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.466787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.467326 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.467327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.467445 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.467929 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:01.467940 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:01.469609 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:01.471373 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:01.471378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:01.471380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:01.471608 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:01.471798 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:01.471798 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:01.471799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:01.471884 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:01.472110 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:01.472116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:01.472269 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 11:13:01.472387 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 11:13:01.472537 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260927 11:13:01.473201 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36307
I20260927 11:13:01.473278 1987 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:01.473459 1987 heartbeater.cc:511] Master 127.0.98.62:36307 requested a full tablet report, sending...
I20260927 11:13:01.473838 1503 ts_manager.cc:194] Registered new tserver with Master: ff39cfe91c9c4452989c52e52a1ad592 (127.0.98.3:35695)
I20260927 11:13:01.474310 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52895
I20260927 11:13:01.478752 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.474265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41979 (local address 127.0.98.62:36307)
0927 11:13:01.474303 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:01.474305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:01.474521 (+ 216us) server_negotiation.cc:407] Connection header received
0927 11:13:01.474583 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:01.474583 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:01.474592 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:01.474634 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:01.474939 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.475360 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.475985 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.476170 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.476213 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.476975 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:01.476976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:01.476976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:01.476980 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:01.477989 (+ 1009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:01.478305 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:01.478305 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:01.478306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:01.478347 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:01.478476 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:01.478476 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:01.478476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:01.478546 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:01.478584 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:01.478660 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 11:13:01.478704 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:13:01.478755 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.474194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36307 (local address 127.0.98.3:41979)
0927 11:13:01.474460 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:01.474471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:01.474529 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:01.474639 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:01.474639 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:01.474823 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:01.474871 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.474873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.475455 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.475456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.475921 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.475922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.476226 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.476227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.476269 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.476663 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:01.476668 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:01.476913 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:01.478099 (+ 1186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:01.478100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:01.478100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:01.478249 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:01.478400 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:01.478400 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:01.478400 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:01.478442 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:01.478577 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:01.478577 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:01.478603 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 11:13:01.478641 (+ 38us) client_negotiation.cc:240] Negotiation successful
0927 11:13:01.478698 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":55,"threads_started":1}
I20260927 11:13:01.484045 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 11:13:01 dist-test-slave-9hq9 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507581, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 11:13:01 dist-test-slave-9hq9 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507581, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:01.513149 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.506292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.0.98.62:36307)
0927 11:13:01.506342 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:01.506344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:01.506353 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:13:01.506379 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:01.506380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:01.506390 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:01.506458 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:01.506997 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.507794 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.508640 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.508831 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.508883 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.511071 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:01.511072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:01.511073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:01.511078 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:01.512171 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:01.512512 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:01.512512 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:01.512513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:01.512571 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:01.512762 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:01.512763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:01.512763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:01.512854 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:01.512932 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:01.513012 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 11:13:01.513064 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 11:13:01.514797 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:13:01.516239 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:01.522770 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.518463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34377 (local address 127.0.0.1:33992)
0927 11:13:01.518756 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:01.518813 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:01.518877 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:01.519259 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:01.519260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:01.519261 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:01.519319 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.519321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.520488 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.520489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.521545 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.521547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.522451 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.522452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.522538 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.522552 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 11:13:01.522633 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 11:13:01.522704 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":42,"thread_start_us":47,"threads_started":1}
I20260927 11:13:01.522794 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.518698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33992 (local address 127.0.98.2:34377)
0927 11:13:01.519030 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:01.519043 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:01.519053 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:13:01.519089 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:01.519091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:01.519105 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:01.519192 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:01.519386 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.520416 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.521630 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.522364 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.522439 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.522506 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:01.522656 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 11:13:01.522705 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260927 11:13:01.522850 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.518527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.0.98.3:35695)
0927 11:13:01.518668 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:01.518673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:01.518875 (+ 202us) server_negotiation.cc:407] Connection header received
0927 11:13:01.518994 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:01.518996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:01.519011 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:01.519084 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:01.519324 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.520409 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.521790 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.522367 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.522423 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.522492 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:01.522716 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 11:13:01.522753 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260927 11:13:01.523036 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.518464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35695 (local address 127.0.0.1:46786)
0927 11:13:01.518756 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:01.518813 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:01.518900 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:01.519168 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:01.519182 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:01.519185 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:01.519237 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.519239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.520513 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.520514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.521691 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.521694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.522549 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.522550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.522616 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.522619 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:01.522916 (+ 297us) client_negotiation.cc:240] Negotiation successful
0927 11:13:01.522977 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":23,"thread_start_us":76,"threads_started":1}
I20260927 11:13:01.523492 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 04d73dd21cd3458f9ee20bf8ebabe246 (DEFAULT_TABLE table=test-table [id=eee0c77d0e8b407fa6a35adcfb8ed028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:01.523841 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d73dd21cd3458f9ee20bf8ebabe246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:01.524348 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.518463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44873 (local address 127.0.0.1:42140)
0927 11:13:01.518862 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:01.518873 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:01.518929 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:01.519257 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:01.519259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:01.519261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:01.519319 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.519321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.521831 (+ 2510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.521832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.523294 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.523312 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.524197 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.524198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.524240 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.524243 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:01.524279 (+ 36us) client_negotiation.cc:240] Negotiation successful
0927 11:13:01.524304 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":191,"thread_start_us":55,"threads_started":1}
I20260927 11:13:01.524422 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.518608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42140 (local address 127.0.98.1:44873)
0927 11:13:01.518906 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:01.518911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:01.518922 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:13:01.519012 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:01.519015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:01.519102 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:01.519192 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:01.520524 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.521755 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.523400 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.524107 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.524198 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.524273 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:01.524297 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 11:13:01.524335 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":74,"threads_started":1}
I20260927 11:13:01.526712 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 04d73dd21cd3458f9ee20bf8ebabe246 (DEFAULT_TABLE table=test-table [id=eee0c77d0e8b407fa6a35adcfb8ed028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:01.526973 2010 tablet_bootstrap.cc:492] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d: Bootstrap starting.
I20260927 11:13:01.527024 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d73dd21cd3458f9ee20bf8ebabe246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:01.527501 2010 tablet_bootstrap.cc:654] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:01.527789 2010 log.cc:824] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:01.528447 2010 tablet_bootstrap.cc:492] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d: No bootstrap required, opened a new log
I20260927 11:13:01.528499 2010 ts_tablet_manager.cc:1397] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:01.530117 2010 raft_consensus.cc:348] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } }
I20260927 11:13:01.530313 2010 raft_consensus.cc:374] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:01.530354 2010 raft_consensus.cc:729] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db82beb4a4344b898bcf19f4e020cd9d, State: Initialized, Role: FOLLOWER
I20260927 11:13:01.530375 2012 tablet_bootstrap.cc:492] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9: Bootstrap starting.
I20260927 11:13:01.530463 2010 consensus_queue.cc:260] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } }
I20260927 11:13:01.530467 1916 tablet_service.cc:1467] Processing CreateTablet for tablet 04d73dd21cd3458f9ee20bf8ebabe246 (DEFAULT_TABLE table=test-table [id=eee0c77d0e8b407fa6a35adcfb8ed028]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:01.530733 1847 heartbeater.cc:503] Master 127.0.98.62:36307 was elected leader, sending a full tablet report...
I20260927 11:13:01.530793 1916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d73dd21cd3458f9ee20bf8ebabe246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:01.530956 2010 ts_tablet_manager.cc:1428] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:13:01.530975 2012 tablet_bootstrap.cc:654] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:01.531263 2012 log.cc:824] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:01.532117 2012 tablet_bootstrap.cc:492] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9: No bootstrap required, opened a new log
I20260927 11:13:01.532157 2012 ts_tablet_manager.cc:1397] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:01.533492 2015 tablet_bootstrap.cc:492] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592: Bootstrap starting.
I20260927 11:13:01.533452 2012 raft_consensus.cc:348] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } }
I20260927 11:13:01.533603 2012 raft_consensus.cc:374] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:01.533635 2012 raft_consensus.cc:729] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7eb1a45adf40ab91828d6a6aad69d9, State: Initialized, Role: FOLLOWER
I20260927 11:13:01.533763 2012 consensus_queue.cc:260] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } }
I20260927 11:13:01.534032 2012 ts_tablet_manager.cc:1428] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:13:01.534152 2015 tablet_bootstrap.cc:654] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:01.534185 1707 heartbeater.cc:503] Master 127.0.98.62:36307 was elected leader, sending a full tablet report...
I20260927 11:13:01.534448 2015 log.cc:824] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:01.535084 2015 tablet_bootstrap.cc:492] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592: No bootstrap required, opened a new log
I20260927 11:13:01.535161 2015 ts_tablet_manager.cc:1397] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 11:13:01.536391 2015 raft_consensus.cc:348] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } }
I20260927 11:13:01.536523 2015 raft_consensus.cc:374] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:01.536569 2015 raft_consensus.cc:729] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff39cfe91c9c4452989c52e52a1ad592, State: Initialized, Role: FOLLOWER
I20260927 11:13:01.536670 2015 consensus_queue.cc:260] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } }
I20260927 11:13:01.536890 2015 ts_tablet_manager.cc:1428] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 11:13:01.536921 1987 heartbeater.cc:503] Master 127.0.98.62:36307 was elected leader, sending a full tablet report...
W20260927 11:13:01.577731 1848 tablet.cc:2369] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:13:01.662380 1708 tablet.cc:2369] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:13:01.719805 1988 tablet.cc:2369] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:13:01.852789 2018 raft_consensus.cc:482] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:01.852906 2018 raft_consensus.cc:504] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } }
I20260927 11:13:01.853240 2018 leader_election.cc:288] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f7eb1a45adf40ab91828d6a6aad69d9 (127.0.98.1:44873), db82beb4a4344b898bcf19f4e020cd9d (127.0.98.2:34377)
I20260927 11:13:01.857075 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.853466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32881 (local address 127.0.98.1:44873)
0927 11:13:01.853520 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:01.853522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:01.853539 (+ 17us) server_negotiation.cc:407] Connection header received
0927 11:13:01.853657 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:01.853658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:01.853669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:01.853738 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:01.853913 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.854706 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.856183 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.856724 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.856799 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.856808 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:01.856970 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 11:13:01.857004 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:13:01.857071 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.853489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43009 (local address 127.0.98.2:34377)
0927 11:13:01.853550 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:01.853552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:01.853705 (+ 153us) server_negotiation.cc:407] Connection header received
0927 11:13:01.853755 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:01.853756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:01.853768 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:01.853820 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:01.853972 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.854706 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.856183 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.856724 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.856799 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.856808 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:01.856970 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 11:13:01.857004 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 11:13:01.857108 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.853433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44873 (local address 127.0.98.3:32881)
0927 11:13:01.853494 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:01.853505 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:01.853552 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:01.853746 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:01.853747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:01.853749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:01.853837 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.853838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.854797 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.854797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.856098 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.856101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.856806 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.856807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.856890 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.856896 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 11:13:01.856953 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 11:13:01.857001 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":66}
I20260927 11:13:01.857318 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.853449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34377 (local address 127.0.98.3:43009)
0927 11:13:01.853604 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:01.853616 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:01.853702 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:01.853837 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:01.853838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:01.853839 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:01.853907 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.853908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.854794 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.854795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.856098 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.856101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.856806 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.856807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.856890 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.856897 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 11:13:01.856953 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 11:13:01.857001 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":69,"thread_start_us":92,"threads_started":1}
I20260927 11:13:01.857460 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d73dd21cd3458f9ee20bf8ebabe246" candidate_uuid: "ff39cfe91c9c4452989c52e52a1ad592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" is_pre_election: true
I20260927 11:13:01.857615 1661 raft_consensus.cc:2454] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff39cfe91c9c4452989c52e52a1ad592 in term 0.
I20260927 11:13:01.857622 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d73dd21cd3458f9ee20bf8ebabe246" candidate_uuid: "ff39cfe91c9c4452989c52e52a1ad592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db82beb4a4344b898bcf19f4e020cd9d" is_pre_election: true
I20260927 11:13:01.857738 1801 raft_consensus.cc:2454] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff39cfe91c9c4452989c52e52a1ad592 in term 0.
I20260927 11:13:01.857829 1874 leader_election.cc:302] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f7eb1a45adf40ab91828d6a6aad69d9, ff39cfe91c9c4452989c52e52a1ad592; no voters:
I20260927 11:13:01.857990 2018 raft_consensus.cc:2781] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:01.858053 2018 raft_consensus.cc:482] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:01.858106 2018 raft_consensus.cc:3037] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:01.859035 2018 raft_consensus.cc:504] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } }
I20260927 11:13:01.859177 2018 leader_election.cc:288] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [CANDIDATE]: Term 1 election: Requested vote from peers 6f7eb1a45adf40ab91828d6a6aad69d9 (127.0.98.1:44873), db82beb4a4344b898bcf19f4e020cd9d (127.0.98.2:34377)
I20260927 11:13:01.859316 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d73dd21cd3458f9ee20bf8ebabe246" candidate_uuid: "ff39cfe91c9c4452989c52e52a1ad592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db82beb4a4344b898bcf19f4e020cd9d"
I20260927 11:13:01.859334 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d73dd21cd3458f9ee20bf8ebabe246" candidate_uuid: "ff39cfe91c9c4452989c52e52a1ad592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9"
I20260927 11:13:01.859416 1801 raft_consensus.cc:3037] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:01.859413 1661 raft_consensus.cc:3037] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:01.860056 1801 raft_consensus.cc:2454] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff39cfe91c9c4452989c52e52a1ad592 in term 1.
I20260927 11:13:01.860106 1661 raft_consensus.cc:2454] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff39cfe91c9c4452989c52e52a1ad592 in term 1.
I20260927 11:13:01.860252 1874 leader_election.cc:302] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f7eb1a45adf40ab91828d6a6aad69d9, ff39cfe91c9c4452989c52e52a1ad592; no voters:
I20260927 11:13:01.860370 2018 raft_consensus.cc:2781] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:01.860539 2018 raft_consensus.cc:686] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [term 1 LEADER]: Becoming Leader. State: Replica: ff39cfe91c9c4452989c52e52a1ad592, State: Running, Role: LEADER
I20260927 11:13:01.860634 2018 consensus_queue.cc:237] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } }
I20260927 11:13:01.861419 1503 catalog_manager.cc:5495] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff39cfe91c9c4452989c52e52a1ad592 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ff39cfe91c9c4452989c52e52a1ad592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } health_report { overall_health: UNKNOWN } } }
I20260927 11:13:01.873888 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.871645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46788 (local address 127.0.98.3:35695)
0927 11:13:01.871695 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:01.871697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:01.871704 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:13:01.871728 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:01.871728 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:01.871742 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:01.871796 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:01.871911 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.872370 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.873136 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.873306 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.873356 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.873570 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:01.873681 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:01.873744 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:01.873787 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 11:13:01.873802 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 11:13:01.875419 1801 raft_consensus.cc:1261] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d [term 1 FOLLOWER]: Refusing update from remote peer ff39cfe91c9c4452989c52e52a1ad592: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:01.875419 1661 raft_consensus.cc:1261] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 1 FOLLOWER]: Refusing update from remote peer ff39cfe91c9c4452989c52e52a1ad592: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:01.875713 2018 consensus_queue.cc:1035] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:01.875831 2020 consensus_queue.cc:1035] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:01.879047 2025 mvcc.cc:205] Tried to move back new op lower bound from 7333919055359311872 to 7333919055301820416. Current Snapshot: MvccSnapshot[applied={T|T < 7333919055359311872 or (T in {7333919055359311872})}]
I20260927 11:13:01.880906 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:01.880996 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:01.881664 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89: Sending DeleteTablet for 3 replicas of tablet 04d73dd21cd3458f9ee20bf8ebabe246
I20260927 11:13:01.881970 1916 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d73dd21cd3458f9ee20bf8ebabe246 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46786
I20260927 11:13:01.882010 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d73dd21cd3458f9ee20bf8ebabe246 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42140
I20260927 11:13:01.882298 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d73dd21cd3458f9ee20bf8ebabe246 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33992
I20260927 11:13:01.882339 2033 tablet_replica.cc:330] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592: stopping tablet replica
I20260927 11:13:01.882429 2033 raft_consensus.cc:2229] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:13:01.882400 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 11:13:01.882512 2033 raft_consensus.cc:2258] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 11:13:01.882534 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:01.882773 2034 tablet_replica.cc:330] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9: stopping tablet replica
I20260927 11:13:01.882872 2034 raft_consensus.cc:2229] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:01.882977 2034 raft_consensus.cc:2258] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:01.883391 2034 ts_tablet_manager.cc:1905] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:01.883858 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 934efb3dc1a5409d96d83aec397f8ccc (DEFAULT_TABLE table=test-table [id=c522c101905f4bbeb614c16cec8e8358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:01.883878 1916 tablet_service.cc:1467] Processing CreateTablet for tablet 934efb3dc1a5409d96d83aec397f8ccc (DEFAULT_TABLE table=test-table [id=c522c101905f4bbeb614c16cec8e8358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:01.883967 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 934efb3dc1a5409d96d83aec397f8ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:01.883975 1916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 934efb3dc1a5409d96d83aec397f8ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:01.885169 2012 tablet_bootstrap.cc:492] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9: Bootstrap starting.
I20260927 11:13:01.885423 2034 ts_tablet_manager.cc:1918] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:01.885481 2034 log.cc:1196] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/wal/wals/04d73dd21cd3458f9ee20bf8ebabe246
I20260927 11:13:01.885540 2012 tablet_bootstrap.cc:654] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:01.885743 2034 ts_tablet_manager.cc:1939] T 04d73dd21cd3458f9ee20bf8ebabe246 P 6f7eb1a45adf40ab91828d6a6aad69d9: Deleting consensus metadata
I20260927 11:13:01.885787 2015 tablet_bootstrap.cc:492] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592: Bootstrap starting.
I20260927 11:13:01.885913 2035 tablet_replica.cc:330] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d: stopping tablet replica
I20260927 11:13:01.886008 2035 raft_consensus.cc:2229] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:01.886149 2035 raft_consensus.cc:2258] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:01.886200 2015 tablet_bootstrap.cc:654] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:01.886535 1490 catalog_manager.cc:4841] TS 6f7eb1a45adf40ab91828d6a6aad69d9 (127.0.98.1:44873): tablet 04d73dd21cd3458f9ee20bf8ebabe246 (table test-table [id=eee0c77d0e8b407fa6a35adcfb8ed028]) successfully deleted
I20260927 11:13:01.886780 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 934efb3dc1a5409d96d83aec397f8ccc (DEFAULT_TABLE table=test-table [id=c522c101905f4bbeb614c16cec8e8358]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:01.886888 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 934efb3dc1a5409d96d83aec397f8ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:01.886929 2012 tablet_bootstrap.cc:492] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9: No bootstrap required, opened a new log
I20260927 11:13:01.886969 2012 ts_tablet_manager.cc:1397] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 11:13:01.887020 2015 tablet_bootstrap.cc:492] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592: No bootstrap required, opened a new log
I20260927 11:13:01.887054 2015 ts_tablet_manager.cc:1397] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:01.887084 2012 raft_consensus.cc:348] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:01.887141 2012 raft_consensus.cc:374] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:01.887161 2012 raft_consensus.cc:729] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7eb1a45adf40ab91828d6a6aad69d9, State: Initialized, Role: FOLLOWER
I20260927 11:13:01.887161 2015 raft_consensus.cc:348] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:01.887208 2015 raft_consensus.cc:374] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:01.887202 2012 consensus_queue.cc:260] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:01.887225 2015 raft_consensus.cc:729] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff39cfe91c9c4452989c52e52a1ad592, State: Initialized, Role: FOLLOWER
I20260927 11:13:01.887272 2015 consensus_queue.cc:260] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:01.887362 2015 ts_tablet_manager.cc:1428] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:01.887558 2012 ts_tablet_manager.cc:1428] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:01.888360 2010 tablet_bootstrap.cc:492] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d: Bootstrap starting.
I20260927 11:13:01.888741 2010 tablet_bootstrap.cc:654] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:01.888842 2035 ts_tablet_manager.cc:1905] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:01.889323 2010 tablet_bootstrap.cc:492] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d: No bootstrap required, opened a new log
I20260927 11:13:01.889367 2010 ts_tablet_manager.cc:1397] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:01.889520 2010 raft_consensus.cc:348] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:01.889578 2010 raft_consensus.cc:374] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:01.889598 2010 raft_consensus.cc:729] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db82beb4a4344b898bcf19f4e020cd9d, State: Initialized, Role: FOLLOWER
I20260927 11:13:01.889640 2010 consensus_queue.cc:260] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:01.889730 2010 ts_tablet_manager.cc:1428] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:01.890153 2035 ts_tablet_manager.cc:1918] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:01.890223 2035 log.cc:1196] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/04d73dd21cd3458f9ee20bf8ebabe246
I20260927 11:13:01.890475 2035 ts_tablet_manager.cc:1939] T 04d73dd21cd3458f9ee20bf8ebabe246 P db82beb4a4344b898bcf19f4e020cd9d: Deleting consensus metadata
I20260927 11:13:01.890698 2033 ts_tablet_manager.cc:1905] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:01.890937 1491 catalog_manager.cc:4841] TS db82beb4a4344b898bcf19f4e020cd9d (127.0.98.2:34377): tablet 04d73dd21cd3458f9ee20bf8ebabe246 (table test-table [id=eee0c77d0e8b407fa6a35adcfb8ed028]) successfully deleted
I20260927 11:13:01.892133 2033 ts_tablet_manager.cc:1918] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:01.892186 2033 log.cc:1196] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/04d73dd21cd3458f9ee20bf8ebabe246
I20260927 11:13:01.892356 2033 ts_tablet_manager.cc:1939] T 04d73dd21cd3458f9ee20bf8ebabe246 P ff39cfe91c9c4452989c52e52a1ad592: Deleting consensus metadata
I20260927 11:13:01.892701 1489 catalog_manager.cc:4841] TS ff39cfe91c9c4452989c52e52a1ad592 (127.0.98.3:35695): tablet 04d73dd21cd3458f9ee20bf8ebabe246 (table test-table [id=eee0c77d0e8b407fa6a35adcfb8ed028]) successfully deleted
I20260927 11:13:01.955423 2013 raft_consensus.cc:482] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:01.955540 2013 raft_consensus.cc:504] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:01.955729 2031 raft_consensus.cc:482] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:01.955804 2031 raft_consensus.cc:504] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:01.955862 2013 leader_election.cc:288] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f7eb1a45adf40ab91828d6a6aad69d9 (127.0.98.1:44873), ff39cfe91c9c4452989c52e52a1ad592 (127.0.98.3:35695)
I20260927 11:13:01.955947 2031 leader_election.cc:288] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f7eb1a45adf40ab91828d6a6aad69d9 (127.0.98.1:44873), db82beb4a4344b898bcf19f4e020cd9d (127.0.98.2:34377)
I20260927 11:13:01.956171 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934efb3dc1a5409d96d83aec397f8ccc" candidate_uuid: "ff39cfe91c9c4452989c52e52a1ad592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" is_pre_election: true
I20260927 11:13:01.956262 1661 raft_consensus.cc:2454] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff39cfe91c9c4452989c52e52a1ad592 in term 0.
I20260927 11:13:01.956362 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934efb3dc1a5409d96d83aec397f8ccc" candidate_uuid: "ff39cfe91c9c4452989c52e52a1ad592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db82beb4a4344b898bcf19f4e020cd9d" is_pre_election: true
I20260927 11:13:01.956393 1874 leader_election.cc:302] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f7eb1a45adf40ab91828d6a6aad69d9, ff39cfe91c9c4452989c52e52a1ad592; no voters:
I20260927 11:13:01.956468 1801 raft_consensus.cc:2454] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff39cfe91c9c4452989c52e52a1ad592 in term 0.
I20260927 11:13:01.956493 2031 raft_consensus.cc:2781] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:01.956522 2031 raft_consensus.cc:482] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:01.956549 2031 raft_consensus.cc:3037] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:01.957468 2031 raft_consensus.cc:504] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:01.957576 2031 leader_election.cc:288] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [CANDIDATE]: Term 1 election: Requested vote from peers 6f7eb1a45adf40ab91828d6a6aad69d9 (127.0.98.1:44873), db82beb4a4344b898bcf19f4e020cd9d (127.0.98.2:34377)
I20260927 11:13:01.957738 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934efb3dc1a5409d96d83aec397f8ccc" candidate_uuid: "ff39cfe91c9c4452989c52e52a1ad592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db82beb4a4344b898bcf19f4e020cd9d"
I20260927 11:13:01.957800 1801 raft_consensus.cc:3037] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:01.958235 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934efb3dc1a5409d96d83aec397f8ccc" candidate_uuid: "ff39cfe91c9c4452989c52e52a1ad592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9"
I20260927 11:13:01.958310 1661 raft_consensus.cc:3037] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:01.958581 1801 raft_consensus.cc:2454] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff39cfe91c9c4452989c52e52a1ad592 in term 1.
I20260927 11:13:01.958758 1875 leader_election.cc:302] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: db82beb4a4344b898bcf19f4e020cd9d, ff39cfe91c9c4452989c52e52a1ad592; no voters:
I20260927 11:13:01.958835 2031 raft_consensus.cc:2781] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:01.958875 2031 raft_consensus.cc:686] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [term 1 LEADER]: Becoming Leader. State: Replica: ff39cfe91c9c4452989c52e52a1ad592, State: Running, Role: LEADER
I20260927 11:13:01.958927 2031 consensus_queue.cc:237] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:01.959095 1661 raft_consensus.cc:2454] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff39cfe91c9c4452989c52e52a1ad592 in term 1.
I20260927 11:13:01.959429 1504 catalog_manager.cc:5495] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff39cfe91c9c4452989c52e52a1ad592 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ff39cfe91c9c4452989c52e52a1ad592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } health_report { overall_health: HEALTHY } } }
I20260927 11:13:01.961397 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.956167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33453 (local address 127.0.98.3:35695)
0927 11:13:01.956207 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:01.956209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:01.956659 (+ 450us) server_negotiation.cc:407] Connection header received
0927 11:13:01.956771 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:01.956772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:01.956781 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:01.956809 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:01.957003 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.957677 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.960702 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.961056 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.961142 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.961151 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:01.961299 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 11:13:01.961338 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:13:01.961501 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.956235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32975 (local address 127.0.98.1:44873)
0927 11:13:01.956276 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:01.956278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:01.956775 (+ 497us) server_negotiation.cc:407] Connection header received
0927 11:13:01.956873 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:01.956874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:01.956882 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:01.956911 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:01.957121 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.957798 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.960600 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.961139 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.961224 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.961235 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:01.961386 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 11:13:01.961432 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:13:01.961416 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.956021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35695 (local address 127.0.98.2:33453)
0927 11:13:01.956453 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:01.956610 (+ 157us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:01.956700 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:01.956866 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:01.956867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:01.956869 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:01.956934 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.956936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.957897 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.957897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.960613 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.960614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.961143 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.961144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.961227 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.961232 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:13:01.961280 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 11:13:01.961320 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":52,"threads_started":1}
I20260927 11:13:01.961702 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:01.956051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44873 (local address 127.0.98.2:32975)
0927 11:13:01.956716 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:01.956726 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:01.956800 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:01.956973 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:01.956973 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:01.956975 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:01.957044 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.957046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.957899 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.957900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.960475 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:01.960477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:01.961238 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:01.961239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:01.961302 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:01.961304 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:13:01.961573 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 11:13:01.961608 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":200,"thread_start_us":93,"threads_started":1}
I20260927 11:13:01.961810 1935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934efb3dc1a5409d96d83aec397f8ccc" candidate_uuid: "db82beb4a4344b898bcf19f4e020cd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff39cfe91c9c4452989c52e52a1ad592" is_pre_election: true
I20260927 11:13:01.962028 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "934efb3dc1a5409d96d83aec397f8ccc" candidate_uuid: "db82beb4a4344b898bcf19f4e020cd9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" is_pre_election: true
I20260927 11:13:01.962145 1661 raft_consensus.cc:2379] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate db82beb4a4344b898bcf19f4e020cd9d in current term 1: Already voted for candidate ff39cfe91c9c4452989c52e52a1ad592 in this term.
I20260927 11:13:01.962319 1734 leader_election.cc:302] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db82beb4a4344b898bcf19f4e020cd9d; no voters: 6f7eb1a45adf40ab91828d6a6aad69d9, ff39cfe91c9c4452989c52e52a1ad592
I20260927 11:13:01.962431 2013 raft_consensus.cc:2726] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 27 11:13:02 dist-test-slave-9hq9 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:02.008937 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:02.002812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57594 (local address 127.0.98.62:36307)
0927 11:13:02.002879 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:02.002881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:02.002970 (+ 89us) server_negotiation.cc:407] Connection header received
0927 11:13:02.003021 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:02.003022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:02.003030 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:02.003086 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:02.003471 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.003942 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.004580 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.004750 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.004797 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:02.006667 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:02.006668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:02.006668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:02.006674 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:02.008057 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:02.008358 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:02.008358 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:02.008359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:02.008405 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:02.008563 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:02.008564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:02.008564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:02.008653 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:02.008702 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:02.008822 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 11:13:02.008879 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 11:13:02.009094 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:02.002725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36307 (local address 127.0.0.1:57594)
0927 11:13:02.002868 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:02.002884 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:02.002969 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:02.003106 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:02.003106 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:02.003366 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:02.003407 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:02.003409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.004004 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.004004 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:02.004516 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:02.004517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.004808 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.004809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:02.004846 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:02.005306 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:02.005310 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:02.006594 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:02.008122 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:02.008127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:02.008128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:02.008297 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:02.008460 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:02.008461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:02.008461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:02.008512 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:02.008706 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:02.008711 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:02.008759 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 11:13:02.008807 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 11:13:02.008948 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":66,"thread_start_us":82,"threads_started":1}
I20260927 11:13:02.010860 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57594:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 11:13:02.011022 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:02.012331 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 38094e1f482c461085933f3b1bb99c2e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c5f55eac10a413eae1b367612cbb3fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:13:02.012372 1916 tablet_service.cc:1467] Processing CreateTablet for tablet 38094e1f482c461085933f3b1bb99c2e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c5f55eac10a413eae1b367612cbb3fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:13:02.012409 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 38094e1f482c461085933f3b1bb99c2e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c5f55eac10a413eae1b367612cbb3fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:13:02.012487 1916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38094e1f482c461085933f3b1bb99c2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:02.012499 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38094e1f482c461085933f3b1bb99c2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:02.012570 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38094e1f482c461085933f3b1bb99c2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:02.013902 2010 tablet_bootstrap.cc:492] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d: Bootstrap starting.
I20260927 11:13:02.013899 2015 tablet_bootstrap.cc:492] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592: Bootstrap starting.
I20260927 11:13:02.014266 2015 tablet_bootstrap.cc:654] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:02.014364 2010 tablet_bootstrap.cc:654] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:02.014504 2012 tablet_bootstrap.cc:492] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9: Bootstrap starting.
I20260927 11:13:02.014853 2012 tablet_bootstrap.cc:654] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:02.014920 2010 tablet_bootstrap.cc:492] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d: No bootstrap required, opened a new log
I20260927 11:13:02.014951 2010 ts_tablet_manager.cc:1397] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:02.015056 2010 raft_consensus.cc:348] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:02.015100 2010 raft_consensus.cc:374] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:02.015112 2010 raft_consensus.cc:729] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db82beb4a4344b898bcf19f4e020cd9d, State: Initialized, Role: FOLLOWER
I20260927 11:13:02.015172 2010 consensus_queue.cc:260] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:02.015236 2015 tablet_bootstrap.cc:492] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592: No bootstrap required, opened a new log
I20260927 11:13:02.015276 2015 ts_tablet_manager.cc:1397] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:02.015290 2010 ts_tablet_manager.cc:1428] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:02.015323 2013 tablet_replica.cc:439] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } }
I20260927 11:13:02.015384 2013 tablet_replica.cc:442] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:02.015367 2015 raft_consensus.cc:348] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:02.015424 2015 raft_consensus.cc:374] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:02.015441 2015 raft_consensus.cc:729] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff39cfe91c9c4452989c52e52a1ad592, State: Initialized, Role: FOLLOWER
I20260927 11:13:02.015503 2015 consensus_queue.cc:260] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:02.015556 2012 tablet_bootstrap.cc:492] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9: No bootstrap required, opened a new log
I20260927 11:13:02.015594 2015 ts_tablet_manager.cc:1428] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:02.015610 2012 ts_tablet_manager.cc:1397] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:02.015609 2020 tablet_replica.cc:439] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } }
I20260927 11:13:02.015678 2020 tablet_replica.cc:442] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:02.015746 2012 raft_consensus.cc:348] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:02.015811 2012 raft_consensus.cc:374] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:02.015830 2012 raft_consensus.cc:729] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7eb1a45adf40ab91828d6a6aad69d9, State: Initialized, Role: FOLLOWER
I20260927 11:13:02.015867 2012 consensus_queue.cc:260] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:02.015981 2012 ts_tablet_manager.cc:1428] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:02.016093 2016 tablet_replica.cc:439] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } }
I20260927 11:13:02.016196 2016 tablet_replica.cc:442] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:02.103866 2016 raft_consensus.cc:482] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:02.103987 2016 raft_consensus.cc:504] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:02.104357 2016 leader_election.cc:288] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db82beb4a4344b898bcf19f4e020cd9d (127.0.98.2:34377), ff39cfe91c9c4452989c52e52a1ad592 (127.0.98.3:35695)
I20260927 11:13:02.107124 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:02.104554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54621 (local address 127.0.98.3:35695)
0927 11:13:02.104597 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:02.104599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:02.104741 (+ 142us) server_negotiation.cc:407] Connection header received
0927 11:13:02.104809 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:02.104810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:02.104823 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:02.104893 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:02.105043 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.105468 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.106420 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.106796 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.106847 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:02.106855 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:02.107014 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 11:13:02.107055 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:13:02.107170 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:02.104570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41165 (local address 127.0.98.2:34377)
0927 11:13:02.104611 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:02.104612 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:02.104768 (+ 156us) server_negotiation.cc:407] Connection header received
0927 11:13:02.104806 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:02.104807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:02.104815 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:02.104885 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:02.105065 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.105475 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.106503 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.106843 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.106911 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:02.106919 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:02.107075 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 11:13:02.107111 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:13:02.107169 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:02.104522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35695 (local address 127.0.98.1:54621)
0927 11:13:02.104665 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:02.104674 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:02.104735 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:02.104898 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:02.104899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:02.104900 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:02.104970 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:02.104972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.105537 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.105538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:02.106342 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:02.106344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.106869 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.106870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:02.106939 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:02.106944 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:13:02.107002 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 11:13:02.107056 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260927 11:13:02.107306 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:02.104527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34377 (local address 127.0.98.1:41165)
0927 11:13:02.104683 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:02.104690 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:02.104761 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:02.104891 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:02.104891 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:02.104893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:02.104986 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:02.104987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.105575 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.105576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:02.106416 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:02.106419 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.106929 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.106929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:02.106997 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:02.107001 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:02.107062 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 11:13:02.107097 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":42,"thread_start_us":93,"threads_started":1}
I20260927 11:13:02.107438 1935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38094e1f482c461085933f3b1bb99c2e" candidate_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff39cfe91c9c4452989c52e52a1ad592" is_pre_election: true
I20260927 11:13:02.107543 1935 raft_consensus.cc:2454] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f7eb1a45adf40ab91828d6a6aad69d9 in term 0.
I20260927 11:13:02.107626 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38094e1f482c461085933f3b1bb99c2e" candidate_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db82beb4a4344b898bcf19f4e020cd9d" is_pre_election: true
I20260927 11:13:02.107698 1801 raft_consensus.cc:2454] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f7eb1a45adf40ab91828d6a6aad69d9 in term 0.
I20260927 11:13:02.107689 1593 leader_election.cc:302] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f7eb1a45adf40ab91828d6a6aad69d9, ff39cfe91c9c4452989c52e52a1ad592; no voters:
I20260927 11:13:02.107800 2016 raft_consensus.cc:2781] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:02.108248 2016 raft_consensus.cc:482] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:02.108280 2016 raft_consensus.cc:3037] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:02.108946 2016 raft_consensus.cc:504] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:02.109081 2016 leader_election.cc:288] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [CANDIDATE]: Term 1 election: Requested vote from peers db82beb4a4344b898bcf19f4e020cd9d (127.0.98.2:34377), ff39cfe91c9c4452989c52e52a1ad592 (127.0.98.3:35695)
I20260927 11:13:02.109220 1935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38094e1f482c461085933f3b1bb99c2e" candidate_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff39cfe91c9c4452989c52e52a1ad592"
I20260927 11:13:02.109257 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38094e1f482c461085933f3b1bb99c2e" candidate_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db82beb4a4344b898bcf19f4e020cd9d"
I20260927 11:13:02.109297 1935 raft_consensus.cc:3037] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:02.109310 1801 raft_consensus.cc:3037] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:02.110039 1801 raft_consensus.cc:2454] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f7eb1a45adf40ab91828d6a6aad69d9 in term 1.
I20260927 11:13:02.110157 1935 raft_consensus.cc:2454] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f7eb1a45adf40ab91828d6a6aad69d9 in term 1.
I20260927 11:13:02.110250 1595 leader_election.cc:302] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f7eb1a45adf40ab91828d6a6aad69d9, db82beb4a4344b898bcf19f4e020cd9d; no voters:
I20260927 11:13:02.110383 2016 raft_consensus.cc:2781] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:02.110548 2016 raft_consensus.cc:686] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 1 LEADER]: Becoming Leader. State: Replica: 6f7eb1a45adf40ab91828d6a6aad69d9, State: Running, Role: LEADER
I20260927 11:13:02.110630 2016 consensus_queue.cc:237] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } }
I20260927 11:13:02.110972 2053 tablet_replica.cc:439] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9: TxnStatusTablet state changed. Reason: New leader 6f7eb1a45adf40ab91828d6a6aad69d9. Latest consensus state: current_term: 1 leader_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } }
I20260927 11:13:02.111052 2053 tablet_replica.cc:442] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:13:02.111285 2055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 11:13:02.111258 1504 catalog_manager.cc:5495] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f7eb1a45adf40ab91828d6a6aad69d9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } health_report { overall_health: UNKNOWN } } }
I20260927 11:13:02.111380 2055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 11:13:02.131491 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:02.128969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42150 (local address 127.0.98.1:44873)
0927 11:13:02.129021 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:02.129023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:02.129055 (+ 32us) server_negotiation.cc:407] Connection header received
0927 11:13:02.129140 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:02.129141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:02.129157 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:02.129214 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:02.129320 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.129771 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.130708 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.130904 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.130984 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:02.131165 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:02.131255 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:02.131334 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:02.131390 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 11:13:02.131407 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 11:13:02.131502 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:02.128924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44873 (local address 127.0.0.1:42150)
0927 11:13:02.128998 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:02.129011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:02.129073 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:02.129215 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:02.129216 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:02.129217 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:13:02.129263 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:02.129265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.129854 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.129855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:02.130616 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:02.130618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.130985 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.130986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:02.131047 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:02.131067 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:13:02.131321 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:13:02.131330 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 11:13:02.131384 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 11:13:02.131428 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 11:13:02.132557 1801 raft_consensus.cc:1261] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d [term 1 FOLLOWER]: Refusing update from remote peer 6f7eb1a45adf40ab91828d6a6aad69d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:02.132597 1935 raft_consensus.cc:1261] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592 [term 1 FOLLOWER]: Refusing update from remote peer 6f7eb1a45adf40ab91828d6a6aad69d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:02.132831 2016 consensus_queue.cc:1035] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:02.132987 2053 consensus_queue.cc:1035] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:02.133379 2013 tablet_replica.cc:439] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d: TxnStatusTablet state changed. Reason: New leader 6f7eb1a45adf40ab91828d6a6aad69d9. Latest consensus state: current_term: 1 leader_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } }
I20260927 11:13:02.133450 2013 tablet_replica.cc:442] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:02.133908 2020 tablet_replica.cc:439] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592: TxnStatusTablet state changed. Reason: New leader 6f7eb1a45adf40ab91828d6a6aad69d9. Latest consensus state: current_term: 1 leader_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } }
I20260927 11:13:02.134006 2020 tablet_replica.cc:442] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:02.134292 2016 tablet_replica.cc:439] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } }
I20260927 11:13:02.134368 2016 tablet_replica.cc:442] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:13:02.134400 2024 mvcc.cc:205] Tried to move back new op lower bound from 7333919056413286400 to 7333919056325677056. Current Snapshot: MvccSnapshot[applied={T|T < 7333919056413286400}]
I20260927 11:13:02.134472 2013 tablet_replica.cc:439] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } }
I20260927 11:13:02.134552 2013 tablet_replica.cc:442] T 38094e1f482c461085933f3b1bb99c2e P db82beb4a4344b898bcf19f4e020cd9d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:02.135116 2053 tablet_replica.cc:439] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } }
I20260927 11:13:02.135187 2053 tablet_replica.cc:442] T 38094e1f482c461085933f3b1bb99c2e P 6f7eb1a45adf40ab91828d6a6aad69d9: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:13:02.135684 2020 tablet_replica.cc:439] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 } } peers { permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 } } peers { permanent_uuid: "ff39cfe91c9c4452989c52e52a1ad592" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35695 } } }
I20260927 11:13:02.135759 2020 tablet_replica.cc:442] T 38094e1f482c461085933f3b1bb99c2e P ff39cfe91c9c4452989c52e52a1ad592: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:02.140811 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:02.138194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60971 (local address 127.0.98.1:44873)
0927 11:13:02.138252 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:02.138254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:02.138355 (+ 101us) server_negotiation.cc:407] Connection header received
0927 11:13:02.138422 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:02.138423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:02.138432 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:02.138494 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:02.138578 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.139280 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.140059 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.140346 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.140421 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:02.140571 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:02.140603 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:02.140672 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:02.140752 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 11:13:02.140769 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 11:13:02.140878 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:02.138146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44873 (local address 127.0.98.3:60971)
0927 11:13:02.138287 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:02.138297 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:02.138355 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:02.138502 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:02.138503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:02.138504 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:13:02.138541 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:02.138542 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.139348 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.139348 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:02.139984 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:02.139986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.140431 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.140432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:02.140484 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:02.140496 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:13:02.140700 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:13:02.140702 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:13:02.140748 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 11:13:02.140796 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":87,"threads_started":1}
I20260927 11:13:02.143091 1661 raft_consensus.cc:1261] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 1 FOLLOWER]: Refusing update from remote peer ff39cfe91c9c4452989c52e52a1ad592: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:02.143095 1801 raft_consensus.cc:1261] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d [term 1 FOLLOWER]: Refusing update from remote peer ff39cfe91c9c4452989c52e52a1ad592: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:02.143265 2031 consensus_queue.cc:1035] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f7eb1a45adf40ab91828d6a6aad69d9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:02.143340 2020 consensus_queue.cc:1035] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db82beb4a4344b898bcf19f4e020cd9d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:02.144802 2023 mvcc.cc:205] Tried to move back new op lower bound from 7333919056456568832 to 7333919055704182784. Current Snapshot: MvccSnapshot[applied={T|T < 7333919056456568832}]
I20260927 11:13:02.150609 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:02.148095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42289 (local address 127.0.98.3:35695)
0927 11:13:02.148133 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:02.148135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:02.148312 (+ 177us) server_negotiation.cc:407] Connection header received
0927 11:13:02.148398 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:02.148399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:02.148407 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:02.148433 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:02.148629 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.149067 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.149795 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.150030 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.150135 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:02.150392 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:02.150425 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:02.150470 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:02.150530 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 11:13:02.150547 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:13:02.150657 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:02.148064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35695 (local address 127.0.98.1:42289)
0927 11:13:02.148265 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:02.148274 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:02.148323 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:02.148510 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:02.148511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:02.148512 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:13:02.148566 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:02.148568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.149141 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.149142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:02.149715 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:02.149717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.150201 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.150202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:02.150268 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:02.150281 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:13:02.150471 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:13:02.150474 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:02.150525 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 11:13:02.150565 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
I20260927 11:13:02.159866 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:02.159952 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:02.160492 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61458cfbbaae4051803d1dfa628b6c89: Sending DeleteTablet for 3 replicas of tablet 934efb3dc1a5409d96d83aec397f8ccc
I20260927 11:13:02.160750 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 934efb3dc1a5409d96d83aec397f8ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33992
I20260927 11:13:02.160763 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 934efb3dc1a5409d96d83aec397f8ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42140
I20260927 11:13:02.160851 1916 tablet_service.cc:1514] Processing DeleteTablet for tablet 934efb3dc1a5409d96d83aec397f8ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46786
I20260927 11:13:02.160856 2034 tablet_replica.cc:330] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9: stopping tablet replica
I20260927 11:13:02.160912 2034 raft_consensus.cc:2229] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:02.160925 2033 tablet_replica.cc:330] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592: stopping tablet replica
I20260927 11:13:02.160953 2034 raft_consensus.cc:2258] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:02.160969 2033 raft_consensus.cc:2229] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:13:02.161015 2033 raft_consensus.cc:2258] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:02.161026 2035 tablet_replica.cc:330] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d: stopping tablet replica
I20260927 11:13:02.161072 2035 raft_consensus.cc:2229] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:02.161108 2035 raft_consensus.cc:2258] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:02.161336 2033 ts_tablet_manager.cc:1905] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:02.161417 2034 ts_tablet_manager.cc:1905] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:02.161440 2035 ts_tablet_manager.cc:1905] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:02.163112 2033 ts_tablet_manager.cc:1918] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:13:02.163118 2035 ts_tablet_manager.cc:1918] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:13:02.163170 2033 log.cc:1196] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/934efb3dc1a5409d96d83aec397f8ccc
I20260927 11:13:02.163172 2035 log.cc:1196] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/934efb3dc1a5409d96d83aec397f8ccc
I20260927 11:13:02.163388 2033 ts_tablet_manager.cc:1939] T 934efb3dc1a5409d96d83aec397f8ccc P ff39cfe91c9c4452989c52e52a1ad592: Deleting consensus metadata
I20260927 11:13:02.163395 2035 ts_tablet_manager.cc:1939] T 934efb3dc1a5409d96d83aec397f8ccc P db82beb4a4344b898bcf19f4e020cd9d: Deleting consensus metadata
I20260927 11:13:02.163453 2034 ts_tablet_manager.cc:1918] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:13:02.163498 2034 log.cc:1196] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790507576023819-392-0/minicluster-data/ts-0/wal/wals/934efb3dc1a5409d96d83aec397f8ccc
I20260927 11:13:02.163695 2034 ts_tablet_manager.cc:1939] T 934efb3dc1a5409d96d83aec397f8ccc P 6f7eb1a45adf40ab91828d6a6aad69d9: Deleting consensus metadata
I20260927 11:13:02.163700 1489 catalog_manager.cc:4841] TS ff39cfe91c9c4452989c52e52a1ad592 (127.0.98.3:35695): tablet 934efb3dc1a5409d96d83aec397f8ccc (table test-table [id=c522c101905f4bbeb614c16cec8e8358]) successfully deleted
I20260927 11:13:02.163738 1491 catalog_manager.cc:4841] TS db82beb4a4344b898bcf19f4e020cd9d (127.0.98.2:34377): tablet 934efb3dc1a5409d96d83aec397f8ccc (table test-table [id=c522c101905f4bbeb614c16cec8e8358]) successfully deleted
I20260927 11:13:02.163975 1490 catalog_manager.cc:4841] TS 6f7eb1a45adf40ab91828d6a6aad69d9 (127.0.98.1:44873): tablet 934efb3dc1a5409d96d83aec397f8ccc (table test-table [id=c522c101905f4bbeb614c16cec8e8358]) successfully deleted
Sep 27 11:13:02 dist-test-slave-9hq9 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507581, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:13:02.172363 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:02.164111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42158 (local address 127.0.98.1:44873)
0927 11:13:02.164146 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:02.164148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:02.164310 (+ 162us) server_negotiation.cc:407] Connection header received
0927 11:13:02.164395 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:02.164396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:02.164404 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:02.164430 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:02.165163 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.165755 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.166506 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.166705 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.166787 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:02.169471 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:02.169478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:02.169479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:02.169486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:02.171208 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:02.171554 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:02.171557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:02.171558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:02.171623 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:02.171820 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:02.171820 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:02.171821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:02.171958 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:02.172100 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:02.172172 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 11:13:02.172249 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 11:13:02.172691 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42158
I20260927 11:13:02.179268 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:02.173571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.0.98.62:36307)
0927 11:13:02.173616 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:02.173618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:02.173653 (+ 35us) server_negotiation.cc:407] Connection header received
0927 11:13:02.173720 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:02.173721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:02.173729 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:02.173799 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:02.174415 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.174870 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.175527 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:02.175737 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:02.175818 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:02.176878 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:02.176879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:02.176879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:02.176886 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:02.178168 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:02.178499 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:02.178500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:02.178500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:02.178552 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:02.178771 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:02.178772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:02.178772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:02.178912 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:02.178994 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:02.179103 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 11:13:02.179173 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 11:13:02.179442 1504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57608
I20260927 11:13:02.180050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 1576
I20260927 11:13:02.186401 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 1716
I20260927 11:13:02.190812 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 1856
I20260927 11:13:02.196130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 1472
2026-09-27T11:13:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2468 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:13:02 dist-test-slave-9hq9 krb5kdc[2075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:13:02 dist-test-slave-9hq9 krb5kdc[2075](info): set up 2 sockets
Sep 27 11:13:02 dist-test-slave-9hq9 krb5kdc[2075](info): commencing operation
krb5kdc: starting...
W20260927 11:13:03.197494 392 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:13:03 dist-test-slave-9hq9 krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:13:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:13:03.304009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41885
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:03.380122 2088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:03.380350 2088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:03.380386 2088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:03.381501 2088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:03.381567 2088 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 11:13:03.381588 2088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:03.381611 2088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:03.381640 2088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:03.381666 2088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:03.383112 2088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41885
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:03.383369 2088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:03.383586 2088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:03.388223 2096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:03.388211 2094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:03.388309 2088 server_base.cc:1034] running on GCE node
W20260927 11:13:03.388223 2093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:03.388708 2088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:03.388936 2088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:03.390102 2088 hybrid_clock.cc:648] HybridClock initialized: now 1790507583390082 us; error 28 us; skew 500 ppm
Sep 27 11:13:03 dist-test-slave-9hq9 krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:03.392424 2088 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:13:03.392870 2088 webserver.cc:460] Webserver started at http://127.0.98.62:37725/ using document root <none> and password file <none>
I20260927 11:13:03.393081 2088 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:03.393124 2088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:03.393268 2088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:03.394335 2088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "f66a8e6b2f254c3eaa83deefe4f696d1"
format_stamp: "Formatted at 2026-09-27 11:13:03 on dist-test-slave-9hq9"
server_key: "a9deef9e89f40f88f5773901f5082417"
server_key_iv: "c0c6213f9f839d0ec88e6eb449cfb45e"
server_key_version: "encryptionkey@0"
I20260927 11:13:03.394673 2088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "f66a8e6b2f254c3eaa83deefe4f696d1"
format_stamp: "Formatted at 2026-09-27 11:13:03 on dist-test-slave-9hq9"
server_key: "a9deef9e89f40f88f5773901f5082417"
server_key_iv: "c0c6213f9f839d0ec88e6eb449cfb45e"
server_key_version: "encryptionkey@0"
I20260927 11:13:03.395889 2088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:03.396551 2103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:03.396741 2088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:03.396823 2088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "f66a8e6b2f254c3eaa83deefe4f696d1"
format_stamp: "Formatted at 2026-09-27 11:13:03 on dist-test-slave-9hq9"
server_key: "a9deef9e89f40f88f5773901f5082417"
server_key_iv: "c0c6213f9f839d0ec88e6eb449cfb45e"
server_key_version: "encryptionkey@0"
I20260927 11:13:03.396872 2088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:03.412670 2088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:03.414368 2088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:03.414474 2088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:03.418737 2088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41885
I20260927 11:13:03.418839 2165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41885 every 8 connection(s)
I20260927 11:13:03.419137 2088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:03.419811 2166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:03.421905 2166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1: Bootstrap starting.
I20260927 11:13:03.422487 2166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:03.422785 2166 log.cc:824] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:03.423477 2166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1: No bootstrap required, opened a new log
I20260927 11:13:03.425441 2166 raft_consensus.cc:348] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66a8e6b2f254c3eaa83deefe4f696d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41885 } }
I20260927 11:13:03.425562 2166 raft_consensus.cc:374] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:03.425606 2166 raft_consensus.cc:729] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66a8e6b2f254c3eaa83deefe4f696d1, State: Initialized, Role: FOLLOWER
I20260927 11:13:03.425781 2166 consensus_queue.cc:260] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66a8e6b2f254c3eaa83deefe4f696d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41885 } }
I20260927 11:13:03.425864 2166 raft_consensus.cc:388] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:03.425926 2166 raft_consensus.cc:482] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:03.425973 2166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:03.426504 2166 raft_consensus.cc:504] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66a8e6b2f254c3eaa83deefe4f696d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41885 } }
I20260927 11:13:03.426609 2166 leader_election.cc:302] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f66a8e6b2f254c3eaa83deefe4f696d1; no voters:
I20260927 11:13:03.426779 2166 leader_election.cc:288] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:03.426803 2169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:03.426965 2169 raft_consensus.cc:686] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [term 1 LEADER]: Becoming Leader. State: Replica: f66a8e6b2f254c3eaa83deefe4f696d1, State: Running, Role: LEADER
I20260927 11:13:03.427047 2166 sys_catalog.cc:564] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:03.427048 2169 consensus_queue.cc:237] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66a8e6b2f254c3eaa83deefe4f696d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41885 } }
I20260927 11:13:03.427462 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f66a8e6b2f254c3eaa83deefe4f696d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66a8e6b2f254c3eaa83deefe4f696d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41885 } } }
I20260927 11:13:03.427471 2170 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f66a8e6b2f254c3eaa83deefe4f696d1. Latest consensus state: current_term: 1 leader_uuid: "f66a8e6b2f254c3eaa83deefe4f696d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66a8e6b2f254c3eaa83deefe4f696d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41885 } } }
I20260927 11:13:03.427583 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:03.427591 2170 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:03.428066 2178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:03.428309 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 2088
I20260927 11:13:03.428418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:03.428534 2178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:03.428594 392 external_mini_cluster.cc:1428] Setting key 83f4c5b4a3de25a2df5d132bdf220e3d
I20260927 11:13:03.430212 2178 catalog_manager.cc:1347] Generated new cluster ID: d54454b9669049bf9bf93dc6c0204e7e
I20260927 11:13:03.430261 2178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:13:03 dist-test-slave-9hq9 krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:03.440348 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.429134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58330 (local address 127.0.98.62:41885)
0927 11:13:03.429570 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:03.429578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:03.429792 (+ 214us) server_negotiation.cc:407] Connection header received
0927 11:13:03.429926 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:03.429933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:03.429984 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:03.430920 (+ 936us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:03.431012 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.432425 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.433427 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.434027 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.434201 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.436885 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:03.436888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:03.436888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:03.436899 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:03.438761 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.439153 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.439155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.439156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.439249 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.439484 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.439485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.439485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.439756 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:03.439879 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:03.440015 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 11:13:03.440132 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":397,"thread_start_us":74,"threads_started":1}
I20260927 11:13:03.453871 2178 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:03.454406 2178 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:03.461145 2178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1: Generated new TSK 0
I20260927 11:13:03.461318 2178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:13:03.485813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41885
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:03.563601 2192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:03.563788 2192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:03.563810 2192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:03.564881 2192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:03.564925 2192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:03.564945 2192 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 11:13:03.564993 2192 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:13:03.565016 2192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:03.566377 2192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:03.566591 2192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:03.566787 2192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:03.571343 2198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:03.571322 2200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:03.571329 2197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:03.571472 2192 server_base.cc:1034] running on GCE node
I20260927 11:13:03.571676 2192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:03.571916 2192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:03.573055 2192 hybrid_clock.cc:648] HybridClock initialized: now 1790507583573035 us; error 34 us; skew 500 ppm
Sep 27 11:13:03 dist-test-slave-9hq9 krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:03.575378 2192 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:13:03.575791 2192 webserver.cc:460] Webserver started at http://127.0.98.1:38365/ using document root <none> and password file <none>
I20260927 11:13:03.575994 2192 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:03.576035 2192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:03.576188 2192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:03.577137 2192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "731470fe975a46b59c0c291190b14a28"
format_stamp: "Formatted at 2026-09-27 11:13:03 on dist-test-slave-9hq9"
server_key: "6743b65dae7d392dd0ba0d218ffd7a1a"
server_key_iv: "8bbac4f291db1b8a25cf05de5389e462"
server_key_version: "encryptionkey@0"
I20260927 11:13:03.577463 2192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "731470fe975a46b59c0c291190b14a28"
format_stamp: "Formatted at 2026-09-27 11:13:03 on dist-test-slave-9hq9"
server_key: "6743b65dae7d392dd0ba0d218ffd7a1a"
server_key_iv: "8bbac4f291db1b8a25cf05de5389e462"
server_key_version: "encryptionkey@0"
I20260927 11:13:03.578714 2192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:13:03.579382 2207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:03.579559 2192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:03.579622 2192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "731470fe975a46b59c0c291190b14a28"
format_stamp: "Formatted at 2026-09-27 11:13:03 on dist-test-slave-9hq9"
server_key: "6743b65dae7d392dd0ba0d218ffd7a1a"
server_key_iv: "8bbac4f291db1b8a25cf05de5389e462"
server_key_version: "encryptionkey@0"
I20260927 11:13:03.579703 2192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:03.594249 2192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:03.595877 2192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:03.596027 2192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:03.596529 2192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:03.596566 2192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:03.596587 2192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:03.596601 2192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 11:13:03 dist-test-slave-9hq9 krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:03.603650 2192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40171
I20260927 11:13:03.603710 2322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40171 every 8 connection(s)
I20260927 11:13:03.604032 2192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:13:03.606172 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.596894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39099 (local address 127.0.98.62:41885)
0927 11:13:03.596950 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:03.596953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:03.597132 (+ 179us) server_negotiation.cc:407] Connection header received
0927 11:13:03.597618 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:03.597620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:03.597632 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:03.597682 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:03.598363 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.598855 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.599991 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.600378 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.600468 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.603593 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:03.603594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:03.603594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:03.603601 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:03.605027 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.605367 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.605368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.605368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.605424 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.605621 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.605622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.605622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.605715 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:03.605763 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:03.605966 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 11:13:03.606026 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 11:13:03.606272 2216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.596879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41885 (local address 127.0.98.1:39099)
0927 11:13:03.597021 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:03.597052 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:03.597505 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:03.597750 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:03.597753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:03.598037 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:03.598261 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.598265 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.598946 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.598947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.599901 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.599903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.600482 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.600483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.600617 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.601132 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:03.601142 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:03.603495 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:03.605092 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:03.605096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:03.605098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:03.605297 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:03.605479 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:03.605480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:03.605480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:03.605567 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:03.605766 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:03.605770 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:03.605874 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 11:13:03.605946 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 11:13:03.606099 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":85,"threads_started":1}
I20260927 11:13:03.606763 2323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41885
I20260927 11:13:03.606848 2323 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:03.607015 2323 heartbeater.cc:511] Master 127.0.98.62:41885 requested a full tablet report, sending...
I20260927 11:13:03.607559 2119 ts_manager.cc:194] Registered new tserver with Master: 731470fe975a46b59c0c291190b14a28 (127.0.98.1:40171)
I20260927 11:13:03.608378 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39099
I20260927 11:13:03.610343 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 2192
I20260927 11:13:03.610417 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:13:03.610589 392 external_mini_cluster.cc:1428] Setting key 4d699c7784571307fa90270ba5d75030
I20260927 11:13:03.613183 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.607708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41885 (local address 127.0.98.1:44767)
0927 11:13:03.607826 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:03.607835 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:03.607898 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:03.608011 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:03.608011 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:03.608143 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:03.608190 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.608191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.608973 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.608974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.609481 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.609483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.609852 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.609853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.609931 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.610327 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:03.610331 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:03.610675 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:03.612158 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:03.612159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:03.612160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:03.612412 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:03.612623 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:03.612623 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:03.612624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:03.612694 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:03.612976 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:03.612976 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:03.613007 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 11:13:03.613057 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 11:13:03.613122 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260927 11:13:03.613261 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.607750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44767 (local address 127.0.98.62:41885)
0927 11:13:03.607788 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:03.607790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:03.607906 (+ 116us) server_negotiation.cc:407] Connection header received
0927 11:13:03.607932 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:03.607932 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:03.607942 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:03.608007 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:03.608247 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.608862 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.609576 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.609771 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.609851 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.610808 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:03.610826 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:03.610826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:03.610832 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:03.612084 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.612485 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.612486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.612486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.612546 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.612761 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.612762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.612762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.612889 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:03.612980 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:03.613081 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 11:13:03.613156 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:13:03.632767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41885
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:03.710529 2332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:03.710777 2332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:03.710815 2332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:03.712330 2332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:03.712401 2332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:03.712445 2332 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 11:13:03.712471 2332 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:13:03.712507 2332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:03.713899 2332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:03.714169 2332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:03.714412 2332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:03.719233 2337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:03.719218 2338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:03.719357 2332 server_base.cc:1034] running on GCE node
W20260927 11:13:03.719259 2340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:03.719628 2332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:03.719863 2332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:03.721033 2332 hybrid_clock.cc:648] HybridClock initialized: now 1790507583721023 us; error 40 us; skew 500 ppm
Sep 27 11:13:03 dist-test-slave-9hq9 krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:03.723402 2332 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:13:03.723839 2332 webserver.cc:460] Webserver started at http://127.0.98.2:37189/ using document root <none> and password file <none>
I20260927 11:13:03.724067 2332 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:03.724133 2332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:03.724270 2332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:03.725211 2332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "7ebb0073746d49d99393458d0d10def7"
format_stamp: "Formatted at 2026-09-27 11:13:03 on dist-test-slave-9hq9"
server_key: "ebfeeaee5fb448c89bb38ee41149eab2"
server_key_iv: "4c69ea5032e88bd0ab9d3c4f7815a816"
server_key_version: "encryptionkey@0"
I20260927 11:13:03.725560 2332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7ebb0073746d49d99393458d0d10def7"
format_stamp: "Formatted at 2026-09-27 11:13:03 on dist-test-slave-9hq9"
server_key: "ebfeeaee5fb448c89bb38ee41149eab2"
server_key_iv: "4c69ea5032e88bd0ab9d3c4f7815a816"
server_key_version: "encryptionkey@0"
I20260927 11:13:03.726945 2332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:13:03.727758 2347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:03.727906 2332 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:03.727988 2332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "7ebb0073746d49d99393458d0d10def7"
format_stamp: "Formatted at 2026-09-27 11:13:03 on dist-test-slave-9hq9"
server_key: "ebfeeaee5fb448c89bb38ee41149eab2"
server_key_iv: "4c69ea5032e88bd0ab9d3c4f7815a816"
server_key_version: "encryptionkey@0"
I20260927 11:13:03.728036 2332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:03.758467 2332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:03.760195 2332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:03.760303 2332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:03.760798 2332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:03.760839 2332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:03.760861 2332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:03.760874 2332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 11:13:03 dist-test-slave-9hq9 krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:03.768011 2332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40711
I20260927 11:13:03.768121 2462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40711 every 8 connection(s)
I20260927 11:13:03.768383 2332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:13:03.770803 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.761200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53959 (local address 127.0.98.62:41885)
0927 11:13:03.761267 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:03.761270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:03.761462 (+ 192us) server_negotiation.cc:407] Connection header received
0927 11:13:03.761933 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:03.761934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:03.761948 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:03.762025 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:03.762637 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.763468 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.764528 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.764714 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.764787 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.767859 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:03.767860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:03.767861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:03.767868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:03.769418 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.769820 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.769821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.769821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.769877 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.770172 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.770173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.770173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.770289 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:03.770377 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:03.770642 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 11:13:03.770715 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 11:13:03.770989 2356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.761170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41885 (local address 127.0.98.2:53959)
0927 11:13:03.761330 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:03.761369 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:03.761824 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:03.762131 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:03.762133 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:03.762356 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:03.762528 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.762531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.763566 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.763567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.764424 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.764427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.764793 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.764794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.764923 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.765439 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:03.765451 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:03.767748 (+ 2297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:03.769517 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:03.769524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:03.769526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:03.769743 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:03.769952 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:03.769953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:03.769954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:03.770107 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:03.770372 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:03.770377 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:03.770510 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 11:13:03.770608 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 11:13:03.770761 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260927 11:13:03.771462 2463 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41885
I20260927 11:13:03.771549 2463 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:03.771731 2463 heartbeater.cc:511] Master 127.0.98.62:41885 requested a full tablet report, sending...
I20260927 11:13:03.772105 2119 ts_manager.cc:194] Registered new tserver with Master: 7ebb0073746d49d99393458d0d10def7 (127.0.98.2:40711)
I20260927 11:13:03.772625 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53959
I20260927 11:13:03.776954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 2332
I20260927 11:13:03.777045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:13:03.777237 392 external_mini_cluster.cc:1428] Setting key c1d4c0c4759e62e2b199a4ce3b63c098
I20260927 11:13:03.777755 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.772492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45473 (local address 127.0.98.62:41885)
0927 11:13:03.772759 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:03.772761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:03.772769 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:13:03.772790 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:03.772791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:03.772800 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:03.772860 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:03.773265 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.773723 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.774490 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.774693 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.774762 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.775508 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:03.775509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:03.775509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:03.775516 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:03.776613 (+ 1097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.776935 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.776936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.776937 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.776986 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.777236 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.777237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.777237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.777423 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:03.777479 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:03.777646 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 11:13:03.777703 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248}
I20260927 11:13:03.777802 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.772427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41885 (local address 127.0.98.2:45473)
0927 11:13:03.772550 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:03.772560 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:03.772617 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:03.772897 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:03.772898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:03.773038 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:03.773085 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.773087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.773814 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.773815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.774378 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.774379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.774773 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.774773 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.774845 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.775165 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:03.775169 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:03.775424 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:03.776703 (+ 1279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:03.776704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:03.776705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:03.776864 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:03.777092 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:03.777093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:03.777093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:03.777165 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:03.777495 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:03.777495 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:03.777538 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 11:13:03.777612 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 11:13:03.777701 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:13:03.800144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41885
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:03.876322 2472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:03.876511 2472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:03.876533 2472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:03.877652 2472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:03.877693 2472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:03.877714 2472 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 11:13:03.877764 2472 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:13:03.877792 2472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:03.879215 2472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:03.879413 2472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:03.879597 2472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:03.884234 2477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:03.884322 2472 server_base.cc:1034] running on GCE node
W20260927 11:13:03.884253 2480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:03.884298 2478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:03.884562 2472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:03.884815 2472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:03.885986 2472 hybrid_clock.cc:648] HybridClock initialized: now 1790507583885974 us; error 40 us; skew 500 ppm
Sep 27 11:13:03 dist-test-slave-9hq9 krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:03.888181 2472 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:13:03.888619 2472 webserver.cc:460] Webserver started at http://127.0.98.3:38069/ using document root <none> and password file <none>
I20260927 11:13:03.888818 2472 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:03.888883 2472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:03.889004 2472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:03.889858 2472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "29bcf25d837b43f2b22019e510e3ac20"
format_stamp: "Formatted at 2026-09-27 11:13:03 on dist-test-slave-9hq9"
server_key: "7e7ddcf181af1d13b1d3515c6d197d9b"
server_key_iv: "0bcc3876282b76e03644f1f04819b018"
server_key_version: "encryptionkey@0"
I20260927 11:13:03.890209 2472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "29bcf25d837b43f2b22019e510e3ac20"
format_stamp: "Formatted at 2026-09-27 11:13:03 on dist-test-slave-9hq9"
server_key: "7e7ddcf181af1d13b1d3515c6d197d9b"
server_key_iv: "0bcc3876282b76e03644f1f04819b018"
server_key_version: "encryptionkey@0"
I20260927 11:13:03.891419 2472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:03.892047 2487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:03.892217 2472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:03.892319 2472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "29bcf25d837b43f2b22019e510e3ac20"
format_stamp: "Formatted at 2026-09-27 11:13:03 on dist-test-slave-9hq9"
server_key: "7e7ddcf181af1d13b1d3515c6d197d9b"
server_key_iv: "0bcc3876282b76e03644f1f04819b018"
server_key_version: "encryptionkey@0"
I20260927 11:13:03.892400 2472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:03.910576 2472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:03.912155 2472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:03.912302 2472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:03.912796 2472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:03.912860 2472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:03.912889 2472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:03.912916 2472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 11:13:03 dist-test-slave-9hq9 krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:03.919798 2472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45633
I20260927 11:13:03.919945 2602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45633 every 8 connection(s)
I20260927 11:13:03.920167 2472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:13:03.921794 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.913145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35711 (local address 127.0.98.62:41885)
0927 11:13:03.913189 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:03.913191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:03.913416 (+ 225us) server_negotiation.cc:407] Connection header received
0927 11:13:03.913880 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:03.913882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:03.913894 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:03.913939 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:03.914602 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.915067 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.916229 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.916512 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.916593 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.919251 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:03.919253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:03.919253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:03.919261 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:03.920826 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.921136 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.921136 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.921137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.921180 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.921338 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.921339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.921339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.921418 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:03.921465 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:03.921683 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 11:13:03.921737 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:13:03.921937 2496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.913131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41885 (local address 127.0.98.3:35711)
0927 11:13:03.913302 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:03.913333 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:03.913782 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:03.914004 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:03.914008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:03.914296 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:03.914504 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.914508 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.915170 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.915171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.916140 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.916142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.916602 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.916603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.916719 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.917214 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:03.917222 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:03.919159 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:03.920891 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:03.920895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:03.920897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:03.921077 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:03.921217 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:03.921218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:03.921218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:03.921293 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:03.921476 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:03.921478 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:03.921588 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 11:13:03.921661 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 11:13:03.921762 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20260927 11:13:03.922389 2603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41885
I20260927 11:13:03.922472 2603 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:03.922631 2603 heartbeater.cc:511] Master 127.0.98.62:41885 requested a full tablet report, sending...
I20260927 11:13:03.923036 2119 ts_manager.cc:194] Registered new tserver with Master: 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3:45633)
I20260927 11:13:03.923548 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35711
I20260927 11:13:03.924072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 2472
I20260927 11:13:03.924180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:13:03.924378 392 external_mini_cluster.cc:1428] Setting key 5457f6dbab8537399bf97b76473357b1
I20260927 11:13:03.925271 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:13:03.929299 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.923329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41885 (local address 127.0.98.3:44589)
0927 11:13:03.923463 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:03.923472 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:03.923531 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:03.923668 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:03.923669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:03.923851 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:03.923917 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.923919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.924751 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.924751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.925422 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.925423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.925762 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.925763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.925833 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.926259 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:03.926263 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:03.926512 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:03.928461 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:03.928462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:03.928462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:03.928607 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:03.928794 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:03.928794 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:03.928795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:03.928838 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:03.929066 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:03.929067 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:03.929110 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 11:13:03.929170 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 11:13:03.929233 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260927 11:13:03.929474 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.923381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44589 (local address 127.0.98.62:41885)
0927 11:13:03.923413 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:03.923414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:03.923516 (+ 102us) server_negotiation.cc:407] Connection header received
0927 11:13:03.923582 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:03.923583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:03.923590 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:03.923661 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:03.924011 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.924637 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.925492 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.925683 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.925760 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.926608 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:03.926610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:03.926610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:03.926616 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:03.928365 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.928677 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.928678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.928678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.928735 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.928873 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.928873 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.928874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.928996 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:03.929252 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:03.929300 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 11:13:03.929380 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
Sep 27 11:13:03 dist-test-slave-9hq9 krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 11:13:03 dist-test-slave-9hq9 krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:03.946697 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.940798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.0.98.62:41885)
0927 11:13:03.940828 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:03.940830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:03.940837 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:13:03.940879 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:03.940880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:03.940887 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:03.940930 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:03.941392 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.941812 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.942517 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.942679 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.942727 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.944564 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:03.944565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:03.944566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:03.944571 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:03.945834 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.946147 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.946148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.946148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.946210 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:03.946354 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:03.946355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:03.946355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:03.946437 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:03.946499 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:03.946577 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 11:13:03.946625 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 11:13:03.948010 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:13:03.949453 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:03.955272 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.951679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49424 (local address 127.0.98.3:45633)
0927 11:13:03.951891 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:03.951895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:03.952180 (+ 285us) server_negotiation.cc:407] Connection header received
0927 11:13:03.952216 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:03.952218 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:03.952232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:03.952300 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:03.952465 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.953411 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.954450 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.954906 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.954988 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.955040 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:03.955162 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 11:13:03.955190 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":65,"threads_started":1}
I20260927 11:13:03.955262 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.951559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45633 (local address 127.0.0.1:49424)
0927 11:13:03.952023 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:03.952033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:03.952091 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:03.952361 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:03.952362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:03.952364 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:03.952398 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.952399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.953494 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.953495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.954373 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.954375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.954991 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.954992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.955064 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.955076 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 11:13:03.955140 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 11:13:03.955187 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":32,"threads_started":1}
I20260927 11:13:03.955425 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.951504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42148 (local address 127.0.98.2:40711)
0927 11:13:03.951672 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:03.951677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:03.951687 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:13:03.951778 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:03.951780 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:03.951796 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:03.951879 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:03.952036 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.952837 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.954273 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.954948 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.955014 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.955063 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:03.955303 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 11:13:03.955339 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20260927 11:13:03.955677 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.951478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.0.98.1:40171)
0927 11:13:03.951780 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:03.951784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:03.951818 (+ 34us) server_negotiation.cc:407] Connection header received
0927 11:13:03.951856 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:03.951858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:03.951883 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:03.951947 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:03.952082 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.953188 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.954488 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.955141 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.955485 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.955549 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:03.955573 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 11:13:03.955615 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":71,"threads_started":1}
I20260927 11:13:03.955415 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.951366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40171 (local address 127.0.0.1:53742)
0927 11:13:03.951526 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:03.951565 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:03.951627 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:03.951956 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:03.951957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:03.951959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:03.952000 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.952002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.953274 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.953275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.954406 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.954408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.955229 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.955230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.955279 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.955282 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:03.955311 (+ 29us) client_negotiation.cc:240] Negotiation successful
0927 11:13:03.955338 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":64,"threads_started":1}
I20260927 11:13:03.955662 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:03.951400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40711 (local address 127.0.0.1:42148)
0927 11:13:03.951622 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:03.951632 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:03.951674 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:03.951895 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:03.951897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:03.951900 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:03.951955 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.951956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.952926 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.952927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.954185 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:03.954187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:03.955096 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:03.955097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:03.955165 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:03.955170 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:13:03.955216 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 11:13:03.955253 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":54,"threads_started":1}
I20260927 11:13:03.956274 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 0b754c08a5cd4289a986aeabec16ba26 (DEFAULT_TABLE table=test-table [id=be4312432d614b2a9f461661fa4e0d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:03.956399 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 0b754c08a5cd4289a986aeabec16ba26 (DEFAULT_TABLE table=test-table [id=be4312432d614b2a9f461661fa4e0d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:03.956629 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b754c08a5cd4289a986aeabec16ba26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:03.956719 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b754c08a5cd4289a986aeabec16ba26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:03.958468 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 0b754c08a5cd4289a986aeabec16ba26 (DEFAULT_TABLE table=test-table [id=be4312432d614b2a9f461661fa4e0d71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:03.958753 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b754c08a5cd4289a986aeabec16ba26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:03.959528 2623 tablet_bootstrap.cc:492] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20: Bootstrap starting.
I20260927 11:13:03.959601 2624 tablet_bootstrap.cc:492] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28: Bootstrap starting.
I20260927 11:13:03.960215 2624 tablet_bootstrap.cc:654] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:03.960304 2623 tablet_bootstrap.cc:654] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:03.960551 2624 log.cc:824] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:03.960620 2623 log.cc:824] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:03.961323 2623 tablet_bootstrap.cc:492] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20: No bootstrap required, opened a new log
I20260927 11:13:03.961396 2623 ts_tablet_manager.cc:1397] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:03.961728 2627 tablet_bootstrap.cc:492] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7: Bootstrap starting.
I20260927 11:13:03.962301 2627 tablet_bootstrap.cc:654] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:03.962420 2624 tablet_bootstrap.cc:492] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28: No bootstrap required, opened a new log
I20260927 11:13:03.962489 2624 ts_tablet_manager.cc:1397] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 11:13:03.962646 2627 log.cc:824] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:03.963282 2627 tablet_bootstrap.cc:492] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7: No bootstrap required, opened a new log
I20260927 11:13:03.963364 2627 ts_tablet_manager.cc:1397] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 11:13:03.963352 2623 raft_consensus.cc:348] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } }
I20260927 11:13:03.963510 2623 raft_consensus.cc:374] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:03.963551 2623 raft_consensus.cc:729] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29bcf25d837b43f2b22019e510e3ac20, State: Initialized, Role: FOLLOWER
I20260927 11:13:03.963649 2623 consensus_queue.cc:260] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } }
I20260927 11:13:03.963915 2623 ts_tablet_manager.cc:1428] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:13:03.963924 2603 heartbeater.cc:503] Master 127.0.98.62:41885 was elected leader, sending a full tablet report...
I20260927 11:13:03.964394 2624 raft_consensus.cc:348] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } }
I20260927 11:13:03.964537 2624 raft_consensus.cc:374] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:03.964578 2624 raft_consensus.cc:729] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731470fe975a46b59c0c291190b14a28, State: Initialized, Role: FOLLOWER
I20260927 11:13:03.964663 2624 consensus_queue.cc:260] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } }
I20260927 11:13:03.964954 2624 ts_tablet_manager.cc:1428] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:13:03.964969 2323 heartbeater.cc:503] Master 127.0.98.62:41885 was elected leader, sending a full tablet report...
I20260927 11:13:03.965391 2627 raft_consensus.cc:348] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } }
I20260927 11:13:03.965552 2627 raft_consensus.cc:374] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:03.965601 2627 raft_consensus.cc:729] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ebb0073746d49d99393458d0d10def7, State: Initialized, Role: FOLLOWER
I20260927 11:13:03.965695 2627 consensus_queue.cc:260] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } }
I20260927 11:13:03.965942 2627 ts_tablet_manager.cc:1428] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 11:13:03.965978 2463 heartbeater.cc:503] Master 127.0.98.62:41885 was elected leader, sending a full tablet report...
W20260927 11:13:04.019209 2464 tablet.cc:2369] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:13:04.104784 2324 tablet.cc:2369] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:13:04.110148 2631 raft_consensus.cc:482] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:04.110241 2631 raft_consensus.cc:504] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } }
I20260927 11:13:04.110611 2631 leader_election.cc:288] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 731470fe975a46b59c0c291190b14a28 (127.0.98.1:40171), 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3:45633)
I20260927 11:13:04.113318 2630 raft_consensus.cc:482] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:04.113451 2630 raft_consensus.cc:504] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } }
I20260927 11:13:04.114432 2356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.110754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40171 (local address 127.0.98.2:44501)
0927 11:13:04.110812 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:04.110825 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:04.110875 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:04.111067 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:04.111068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:04.111069 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:04.111141 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.111143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.112072 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.112073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.113402 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.113405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.114186 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.114187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.114268 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.114272 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:04.114305 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 11:13:04.114343 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 11:13:04.113883 2630 leader_election.cc:288] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3:45633), 7ebb0073746d49d99393458d0d10def7 (127.0.98.2:40711)
I20260927 11:13:04.114711 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.110981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56897 (local address 127.0.98.3:45633)
0927 11:13:04.111023 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:04.111025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:04.111108 (+ 83us) server_negotiation.cc:407] Connection header received
0927 11:13:04.111187 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:04.111188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:04.111200 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:04.111258 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:04.111431 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.112123 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.113633 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.114269 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.114318 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.114327 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:04.114576 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 11:13:04.114623 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:13:04.114809 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.110826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44501 (local address 127.0.98.1:40171)
0927 11:13:04.110888 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:04.110890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:04.110902 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:13:04.110936 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:04.110937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:04.110948 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:04.111050 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:04.111210 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.111940 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.113510 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.114091 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.114666 (+ 575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.114677 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:04.114694 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 11:13:04.114742 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 11:13:04.114984 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.110871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45633 (local address 127.0.98.2:56897)
0927 11:13:04.111035 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:04.111045 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:04.111097 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:04.111283 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:04.111284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:04.111286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:04.111363 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.111365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.112216 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.112217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.113523 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.113526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.114393 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.114395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.114483 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.114487 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:04.114816 (+ 329us) client_negotiation.cc:240] Negotiation successful
0927 11:13:04.114863 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":1,"thread_start_us":124,"threads_started":1}
I20260927 11:13:04.115037 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b754c08a5cd4289a986aeabec16ba26" candidate_uuid: "7ebb0073746d49d99393458d0d10def7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731470fe975a46b59c0c291190b14a28" is_pre_election: true
I20260927 11:13:04.115134 2277 raft_consensus.cc:2454] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ebb0073746d49d99393458d0d10def7 in term 0.
I20260927 11:13:04.115370 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b754c08a5cd4289a986aeabec16ba26" candidate_uuid: "7ebb0073746d49d99393458d0d10def7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bcf25d837b43f2b22019e510e3ac20" is_pre_election: true
I20260927 11:13:04.115500 2557 raft_consensus.cc:2454] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ebb0073746d49d99393458d0d10def7 in term 0.
I20260927 11:13:04.115736 2349 leader_election.cc:302] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731470fe975a46b59c0c291190b14a28, 7ebb0073746d49d99393458d0d10def7; no voters:
I20260927 11:13:04.115950 2631 raft_consensus.cc:2781] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:04.115993 2631 raft_consensus.cc:482] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:04.116016 2631 raft_consensus.cc:3037] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:04.118732 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.114431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45633 (local address 127.0.98.1:44497)
0927 11:13:04.115211 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:04.115219 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:04.115262 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:04.115435 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:04.115436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:04.115438 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:04.115491 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.115493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.116307 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.116308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.117525 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.117528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.118488 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.118490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.118561 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.118566 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:13:04.118620 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 11:13:04.118656 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":35,"threads_started":1}
I20260927 11:13:04.118790 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.114548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33179 (local address 127.0.98.2:40711)
0927 11:13:04.114617 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:04.114619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:04.115080 (+ 461us) server_negotiation.cc:407] Connection header received
0927 11:13:04.115191 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:04.115192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:04.115202 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:04.115252 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:04.115426 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.116134 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.117640 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.118401 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.118453 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.118463 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:04.118688 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 11:13:04.118727 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 11:13:04.118816 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.114108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44497 (local address 127.0.98.3:45633)
0927 11:13:04.114602 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:04.114604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:04.115311 (+ 707us) server_negotiation.cc:407] Connection header received
0927 11:13:04.115330 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:04.115330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:04.115339 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:04.115398 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:04.115563 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.116217 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.117637 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.118349 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.118440 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.118454 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:04.118631 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 11:13:04.118664 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":95,"threads_started":1}
I20260927 11:13:04.119026 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.114323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40711 (local address 127.0.98.1:33179)
0927 11:13:04.115016 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:04.115026 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:04.115106 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:04.115295 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:04.115297 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:04.115300 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:04.115342 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.115343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.116334 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.116335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.117536 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.117539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.118524 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.118525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.118598 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.118602 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:04.118871 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 11:13:04.118921 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":15,"thread_start_us":72,"threads_started":1}
I20260927 11:13:04.119022 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b754c08a5cd4289a986aeabec16ba26" candidate_uuid: "731470fe975a46b59c0c291190b14a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bcf25d837b43f2b22019e510e3ac20" is_pre_election: true
I20260927 11:13:04.119089 2557 raft_consensus.cc:2454] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731470fe975a46b59c0c291190b14a28 in term 0.
I20260927 11:13:04.119258 2209 leader_election.cc:302] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29bcf25d837b43f2b22019e510e3ac20, 731470fe975a46b59c0c291190b14a28; no voters:
I20260927 11:13:04.119374 2630 raft_consensus.cc:2781] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:04.119410 2630 raft_consensus.cc:482] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:04.119429 2630 raft_consensus.cc:3037] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:04.119445 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b754c08a5cd4289a986aeabec16ba26" candidate_uuid: "731470fe975a46b59c0c291190b14a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ebb0073746d49d99393458d0d10def7" is_pre_election: true
I20260927 11:13:04.166771 2630 raft_consensus.cc:504] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } }
I20260927 11:13:04.166771 2631 raft_consensus.cc:504] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } }
I20260927 11:13:04.166954 2630 leader_election.cc:288] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [CANDIDATE]: Term 1 election: Requested vote from peers 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3:45633), 7ebb0073746d49d99393458d0d10def7 (127.0.98.2:40711)
I20260927 11:13:04.166967 2417 raft_consensus.cc:2379] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 731470fe975a46b59c0c291190b14a28 in current term 1: Already voted for candidate 7ebb0073746d49d99393458d0d10def7 in this term.
I20260927 11:13:04.167239 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b754c08a5cd4289a986aeabec16ba26" candidate_uuid: "731470fe975a46b59c0c291190b14a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ebb0073746d49d99393458d0d10def7"
I20260927 11:13:04.167271 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b754c08a5cd4289a986aeabec16ba26" candidate_uuid: "731470fe975a46b59c0c291190b14a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bcf25d837b43f2b22019e510e3ac20"
I20260927 11:13:04.167344 2416 raft_consensus.cc:2379] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 731470fe975a46b59c0c291190b14a28 in current term 1: Already voted for candidate 7ebb0073746d49d99393458d0d10def7 in this term.
I20260927 11:13:04.167373 2557 raft_consensus.cc:3037] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:04.167430 2631 leader_election.cc:288] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [CANDIDATE]: Term 1 election: Requested vote from peers 731470fe975a46b59c0c291190b14a28 (127.0.98.1:40171), 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3:45633)
I20260927 11:13:04.167657 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b754c08a5cd4289a986aeabec16ba26" candidate_uuid: "7ebb0073746d49d99393458d0d10def7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bcf25d837b43f2b22019e510e3ac20"
I20260927 11:13:04.167974 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b754c08a5cd4289a986aeabec16ba26" candidate_uuid: "7ebb0073746d49d99393458d0d10def7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731470fe975a46b59c0c291190b14a28"
I20260927 11:13:04.168092 2277 raft_consensus.cc:2379] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7ebb0073746d49d99393458d0d10def7 in current term 1: Already voted for candidate 731470fe975a46b59c0c291190b14a28 in this term.
I20260927 11:13:04.168306 2349 leader_election.cc:302] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7ebb0073746d49d99393458d0d10def7; no voters: 29bcf25d837b43f2b22019e510e3ac20, 731470fe975a46b59c0c291190b14a28
I20260927 11:13:04.168447 2631 raft_consensus.cc:2726] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 11:13:04.168737 2557 raft_consensus.cc:2454] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731470fe975a46b59c0c291190b14a28 in term 1.
I20260927 11:13:04.168881 2209 leader_election.cc:302] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29bcf25d837b43f2b22019e510e3ac20, 731470fe975a46b59c0c291190b14a28; no voters: 7ebb0073746d49d99393458d0d10def7
I20260927 11:13:04.168987 2630 raft_consensus.cc:2781] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:04.169101 2630 raft_consensus.cc:686] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [term 1 LEADER]: Becoming Leader. State: Replica: 731470fe975a46b59c0c291190b14a28, State: Running, Role: LEADER
I20260927 11:13:04.169214 2630 consensus_queue.cc:237] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } }
I20260927 11:13:04.170128 2119 catalog_manager.cc:5495] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 731470fe975a46b59c0c291190b14a28 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "731470fe975a46b59c0c291190b14a28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } health_report { overall_health: UNKNOWN } } }
W20260927 11:13:04.170668 2604 tablet.cc:2369] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 11:13:04 dist-test-slave-9hq9 krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:04.205317 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.198703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41885 (local address 127.0.0.1:58344)
0927 11:13:04.198874 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:04.198884 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:04.198964 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:04.199105 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:04.199106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:04.199415 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:04.199455 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.199458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.200136 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.200137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.200658 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.200659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.200997 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.200998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.201052 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.201589 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:04.201593 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:04.202927 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:04.204462 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:04.204467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:04.204469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:04.204642 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:04.204807 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:04.204808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:04.204808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:04.204861 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:04.205058 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:04.205063 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:04.205103 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 11:13:04.205153 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 11:13:04.205231 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":96,"threads_started":1}
I20260927 11:13:04.205407 2188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.198733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.0.98.62:41885)
0927 11:13:04.198786 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:04.198788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:04.198946 (+ 158us) server_negotiation.cc:407] Connection header received
0927 11:13:04.199015 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:04.199016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:04.199026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:04.199088 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:04.199541 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.200053 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.200735 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.200913 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.200996 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.203023 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:04.203024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:04.203025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:04.203030 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:04.204392 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:04.204700 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:04.204700 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:04.204701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:04.204746 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:04.204915 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:04.204916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:04.204916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:04.205003 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:04.205055 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:04.205167 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 11:13:04.205273 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":29}
I20260927 11:13:04.207296 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58344:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 11:13:04.207541 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:04.208927 2257 tablet_service.cc:1467] Processing CreateTablet for tablet c1073007d9ad44a3bfb628e020f9392f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ada615ea1134308b3de8ed14442e475]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:13:04.208963 2397 tablet_service.cc:1467] Processing CreateTablet for tablet c1073007d9ad44a3bfb628e020f9392f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ada615ea1134308b3de8ed14442e475]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:13:04.209091 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1073007d9ad44a3bfb628e020f9392f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:04.209091 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1073007d9ad44a3bfb628e020f9392f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:04.209002 2537 tablet_service.cc:1467] Processing CreateTablet for tablet c1073007d9ad44a3bfb628e020f9392f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5ada615ea1134308b3de8ed14442e475]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:13:04.209229 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1073007d9ad44a3bfb628e020f9392f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:04.210361 2623 tablet_bootstrap.cc:492] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20: Bootstrap starting.
I20260927 11:13:04.210546 2627 tablet_bootstrap.cc:492] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7: Bootstrap starting.
I20260927 11:13:04.210562 2624 tablet_bootstrap.cc:492] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28: Bootstrap starting.
I20260927 11:13:04.210772 2623 tablet_bootstrap.cc:654] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:04.210953 2624 tablet_bootstrap.cc:654] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:04.210953 2627 tablet_bootstrap.cc:654] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:04.211668 2627 tablet_bootstrap.cc:492] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7: No bootstrap required, opened a new log
I20260927 11:13:04.211725 2627 ts_tablet_manager.cc:1397] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:04.211872 2627 raft_consensus.cc:348] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.211925 2627 raft_consensus.cc:374] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:04.211939 2627 raft_consensus.cc:729] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ebb0073746d49d99393458d0d10def7, State: Initialized, Role: FOLLOWER
I20260927 11:13:04.211911 2623 tablet_bootstrap.cc:492] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20: No bootstrap required, opened a new log
I20260927 11:13:04.211970 2623 ts_tablet_manager.cc:1397] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:04.211992 2627 consensus_queue.cc:260] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.212095 2627 ts_tablet_manager.cc:1428] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:04.212114 2623 raft_consensus.cc:348] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.212184 2623 raft_consensus.cc:374] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:04.212210 2623 raft_consensus.cc:729] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29bcf25d837b43f2b22019e510e3ac20, State: Initialized, Role: FOLLOWER
I20260927 11:13:04.212246 2623 consensus_queue.cc:260] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.212325 2623 ts_tablet_manager.cc:1428] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:04.212399 2629 tablet_replica.cc:439] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } }
I20260927 11:13:04.212419 2631 tablet_replica.cc:439] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } }
I20260927 11:13:04.212474 2629 tablet_replica.cc:442] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:04.212486 2631 tablet_replica.cc:442] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:04.212541 2624 tablet_bootstrap.cc:492] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28: No bootstrap required, opened a new log
I20260927 11:13:04.212574 2624 ts_tablet_manager.cc:1397] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:04.212706 2624 raft_consensus.cc:348] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.212770 2624 raft_consensus.cc:374] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:04.212805 2624 raft_consensus.cc:729] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731470fe975a46b59c0c291190b14a28, State: Initialized, Role: FOLLOWER
I20260927 11:13:04.212862 2624 consensus_queue.cc:260] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.212981 2624 ts_tablet_manager.cc:1428] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:04.213034 2630 tablet_replica.cc:439] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } }
I20260927 11:13:04.213150 2630 tablet_replica.cc:442] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:04.463490 2629 raft_consensus.cc:482] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:04.463593 2629 raft_consensus.cc:504] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.464232 2629 leader_election.cc:288] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 731470fe975a46b59c0c291190b14a28 (127.0.98.1:40171), 7ebb0073746d49d99393458d0d10def7 (127.0.98.2:40711)
I20260927 11:13:04.467738 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.464378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40711 (local address 127.0.98.3:35803)
0927 11:13:04.464589 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:04.464599 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:04.464657 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:04.464834 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:04.464834 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:04.464836 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:04.464879 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.464880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.465708 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.465709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.466625 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.466627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.467469 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.467470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.467546 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.467550 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:04.467606 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 11:13:04.467656 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":2,"thread_start_us":51,"threads_started":1}
I20260927 11:13:04.467765 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.464533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35803 (local address 127.0.98.2:40711)
0927 11:13:04.464578 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:04.464581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:04.464641 (+ 60us) server_negotiation.cc:407] Connection header received
0927 11:13:04.464737 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:04.464738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:04.464747 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:04.464775 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:04.464932 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.465616 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.466717 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.467378 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.467468 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.467478 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:04.467645 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 11:13:04.467686 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:13:04.468017 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1073007d9ad44a3bfb628e020f9392f" candidate_uuid: "29bcf25d837b43f2b22019e510e3ac20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ebb0073746d49d99393458d0d10def7" is_pre_election: true
I20260927 11:13:04.468123 2416 raft_consensus.cc:2454] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29bcf25d837b43f2b22019e510e3ac20 in term 0.
I20260927 11:13:04.468340 2491 leader_election.cc:302] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29bcf25d837b43f2b22019e510e3ac20, 7ebb0073746d49d99393458d0d10def7; no voters:
I20260927 11:13:04.469978 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.464378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40171 (local address 127.0.98.3:34477)
0927 11:13:04.464494 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:04.464508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:04.464578 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:04.464743 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:04.464744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:04.464747 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:04.464834 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.464837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.467981 (+ 3144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.467982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.468876 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.468878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.469707 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.469707 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.469779 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.469783 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:04.469869 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 11:13:04.469905 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"spinlock_wait_cycles":896,"thread_start_us":70,"threads_started":1}
I20260927 11:13:04.469978 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.464587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34477 (local address 127.0.98.1:40171)
0927 11:13:04.464635 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:04.464637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:04.464645 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:13:04.464674 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:04.464675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:04.464684 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:04.464741 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:04.464902 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.467894 (+ 2992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.468966 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.469621 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.469704 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.469714 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:04.469879 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 11:13:04.469920 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 11:13:04.470140 2629 raft_consensus.cc:2781] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:04.470196 2629 raft_consensus.cc:482] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:04.470214 2629 raft_consensus.cc:3037] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:04.470220 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1073007d9ad44a3bfb628e020f9392f" candidate_uuid: "29bcf25d837b43f2b22019e510e3ac20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731470fe975a46b59c0c291190b14a28" is_pre_election: true
I20260927 11:13:04.470311 2277 raft_consensus.cc:2454] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29bcf25d837b43f2b22019e510e3ac20 in term 0.
I20260927 11:13:04.496611 2629 raft_consensus.cc:504] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.496788 2629 leader_election.cc:288] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [CANDIDATE]: Term 1 election: Requested vote from peers 731470fe975a46b59c0c291190b14a28 (127.0.98.1:40171), 7ebb0073746d49d99393458d0d10def7 (127.0.98.2:40711)
I20260927 11:13:04.496970 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1073007d9ad44a3bfb628e020f9392f" candidate_uuid: "29bcf25d837b43f2b22019e510e3ac20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731470fe975a46b59c0c291190b14a28"
I20260927 11:13:04.497064 2277 raft_consensus.cc:3037] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:04.497033 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1073007d9ad44a3bfb628e020f9392f" candidate_uuid: "29bcf25d837b43f2b22019e510e3ac20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ebb0073746d49d99393458d0d10def7"
I20260927 11:13:04.497107 2416 raft_consensus.cc:3037] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:04.497887 2416 raft_consensus.cc:2454] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29bcf25d837b43f2b22019e510e3ac20 in term 1.
I20260927 11:13:04.497888 2277 raft_consensus.cc:2454] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29bcf25d837b43f2b22019e510e3ac20 in term 1.
I20260927 11:13:04.498090 2491 leader_election.cc:302] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29bcf25d837b43f2b22019e510e3ac20, 7ebb0073746d49d99393458d0d10def7; no voters:
I20260927 11:13:04.498298 2629 raft_consensus.cc:2781] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:04.498421 2629 raft_consensus.cc:686] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [term 1 LEADER]: Becoming Leader. State: Replica: 29bcf25d837b43f2b22019e510e3ac20, State: Running, Role: LEADER
I20260927 11:13:04.498523 2629 consensus_queue.cc:237] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.499076 2649 tablet_replica.cc:439] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20: TxnStatusTablet state changed. Reason: New leader 29bcf25d837b43f2b22019e510e3ac20. Latest consensus state: current_term: 1 leader_uuid: "29bcf25d837b43f2b22019e510e3ac20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } }
I20260927 11:13:04.499214 2649 tablet_replica.cc:442] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:13:04.499431 2651 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 11:13:04.499553 2651 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 11:13:04.499481 2119 catalog_manager.cc:5495] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "29bcf25d837b43f2b22019e510e3ac20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } health_report { overall_health: HEALTHY } } }
I20260927 11:13:04.566419 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.563983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.0.98.3:45633)
0927 11:13:04.564036 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:04.564039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:04.564048 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:13:04.564109 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:04.564110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:04.564126 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:04.564186 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:04.564311 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.564773 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.565551 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.565784 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.565870 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.566071 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:04.566167 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:04.566257 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:04.566302 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 11:13:04.566321 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 11:13:04.566421 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.563901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45633 (local address 127.0.0.1:38082)
0927 11:13:04.563976 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:04.563988 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:04.564035 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:04.564188 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:04.564189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:04.564190 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:13:04.564244 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.564246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.564877 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.564878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.565460 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.565462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.565874 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.565875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.565946 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.565963 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:13:04.566237 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:13:04.566245 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 11:13:04.566290 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 11:13:04.566348 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 11:13:04.568140 2277 raft_consensus.cc:1261] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28 [term 1 FOLLOWER]: Refusing update from remote peer 29bcf25d837b43f2b22019e510e3ac20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:04.568152 2416 raft_consensus.cc:1261] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Refusing update from remote peer 29bcf25d837b43f2b22019e510e3ac20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:04.568416 2649 consensus_queue.cc:1035] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:04.568583 2629 consensus_queue.cc:1035] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:04.569788 2630 tablet_replica.cc:439] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28: TxnStatusTablet state changed. Reason: New leader 29bcf25d837b43f2b22019e510e3ac20. Latest consensus state: current_term: 1 leader_uuid: "29bcf25d837b43f2b22019e510e3ac20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } }
I20260927 11:13:04.569873 2630 tablet_replica.cc:442] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:04.570950 2631 tablet_replica.cc:439] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7: TxnStatusTablet state changed. Reason: New leader 29bcf25d837b43f2b22019e510e3ac20. Latest consensus state: current_term: 1 leader_uuid: "29bcf25d837b43f2b22019e510e3ac20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } }
I20260927 11:13:04.571046 2631 tablet_replica.cc:442] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:04.571390 2629 tablet_replica.cc:439] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29bcf25d837b43f2b22019e510e3ac20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } }
I20260927 11:13:04.571465 2654 mvcc.cc:205] Tried to move back new op lower bound from 7333919066388811776 to 7333919066106630144. Current Snapshot: MvccSnapshot[applied={T|T < 7333919066388811776}]
I20260927 11:13:04.571878 2649 tablet_replica.cc:439] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29bcf25d837b43f2b22019e510e3ac20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } }
I20260927 11:13:04.571949 2649 tablet_replica.cc:442] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:13:04.572013 2630 tablet_replica.cc:439] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29bcf25d837b43f2b22019e510e3ac20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } }
I20260927 11:13:04.572098 2630 tablet_replica.cc:442] T c1073007d9ad44a3bfb628e020f9392f P 731470fe975a46b59c0c291190b14a28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:04.572495 2631 tablet_replica.cc:439] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29bcf25d837b43f2b22019e510e3ac20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } } }
I20260927 11:13:04.572572 2631 tablet_replica.cc:442] T c1073007d9ad44a3bfb628e020f9392f P 7ebb0073746d49d99393458d0d10def7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:04.573027 2657 mvcc.cc:205] Tried to move back new op lower bound from 7333919066388811776 to 7333919066106630144. Current Snapshot: MvccSnapshot[applied={T|T < 7333919066388811776}]
I20260927 11:13:04.571481 2629 tablet_replica.cc:442] T c1073007d9ad44a3bfb628e020f9392f P 29bcf25d837b43f2b22019e510e3ac20: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:13:04.576668 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.573963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44088 (local address 127.0.98.1:40171)
0927 11:13:04.574004 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:04.574006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:04.574198 (+ 192us) server_negotiation.cc:407] Connection header received
0927 11:13:04.574276 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:04.574276 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:04.574288 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:04.574345 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:04.574481 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.574894 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.576007 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.576178 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.576228 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.576405 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:04.576480 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:04.576528 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:04.576608 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 11:13:04.576624 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:13:04.581422 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.578844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45633 (local address 127.0.98.1:35451)
0927 11:13:04.579023 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:04.579031 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:04.579077 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:04.579210 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:04.579210 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:04.579211 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:13:04.579249 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.579250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.580054 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.580054 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.580580 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.580582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.581052 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.581053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.581113 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.581123 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:13:04.581297 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:13:04.581298 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 11:13:04.581331 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 11:13:04.581361 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
I20260927 11:13:04.581418 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.578915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35451 (local address 127.0.98.3:45633)
0927 11:13:04.578955 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:04.578957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:04.579118 (+ 161us) server_negotiation.cc:407] Connection header received
0927 11:13:04.579140 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:04.579141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:04.579150 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:04.579208 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:04.579289 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.579979 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.580661 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.580978 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.581050 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.581202 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:04.581234 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:04.581291 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:04.581334 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 11:13:04.581353 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:13:04.583238 2416 raft_consensus.cc:1261] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Refusing update from remote peer 731470fe975a46b59c0c291190b14a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:04.583343 2557 raft_consensus.cc:1261] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Refusing update from remote peer 731470fe975a46b59c0c291190b14a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:04.583510 2636 consensus_queue.cc:1035] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:04.583639 2630 consensus_queue.cc:1035] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:04.585135 2655 mvcc.cc:205] Tried to move back new op lower bound from 7333919066451447808 to 7333919064757755904. Current Snapshot: MvccSnapshot[applied={T|T < 7333919066451447808}]
I20260927 11:13:04.592633 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.589379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37413 (local address 127.0.98.1:40171)
0927 11:13:04.589424 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:04.589425 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:04.590254 (+ 829us) server_negotiation.cc:407] Connection header received
0927 11:13:04.590330 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:04.590331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:04.590340 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:04.590387 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:04.590501 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.591075 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.591994 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.592197 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.592269 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.592437 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:04.592465 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:04.592506 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:04.592560 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 11:13:04.592576 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:13:04.592684 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:04.589192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40171 (local address 127.0.98.3:37413)
0927 11:13:04.590191 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:04.590200 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:04.590267 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:04.590389 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:04.590390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:04.590391 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:13:04.590441 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.590442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.591168 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.591169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.591914 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:04.591916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:04.592269 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:04.592270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:04.592331 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:04.592343 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:13:04.592518 (+ 175us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:13:04.592519 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 11:13:04.592560 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 11:13:04.592600 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"thread_start_us":79,"threads_started":1}
I20260927 11:13:04.600793 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:04.600878 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:04.601471 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1: Sending DeleteTablet for 3 replicas of tablet 0b754c08a5cd4289a986aeabec16ba26
I20260927 11:13:04.601742 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b754c08a5cd4289a986aeabec16ba26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49424
I20260927 11:13:04.601887 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b754c08a5cd4289a986aeabec16ba26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42148
I20260927 11:13:04.601933 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b754c08a5cd4289a986aeabec16ba26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53742
I20260927 11:13:04.602118 2676 tablet_replica.cc:330] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7: stopping tablet replica
I20260927 11:13:04.602119 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 11:13:04.602213 2676 raft_consensus.cc:2229] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 11:13:04.602243 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:04.602272 2676 raft_consensus.cc:2258] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:04.603298 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 559ce6f9e5ad49409a38b3cfedd2eaea (DEFAULT_TABLE table=test-table [id=e26becdf135249f7a55ef9ddb48888ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:04.603312 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 559ce6f9e5ad49409a38b3cfedd2eaea (DEFAULT_TABLE table=test-table [id=e26becdf135249f7a55ef9ddb48888ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:04.603406 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559ce6f9e5ad49409a38b3cfedd2eaea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:04.603441 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559ce6f9e5ad49409a38b3cfedd2eaea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:04.604694 2627 tablet_bootstrap.cc:492] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7: Bootstrap starting.
I20260927 11:13:04.604810 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 559ce6f9e5ad49409a38b3cfedd2eaea (DEFAULT_TABLE table=test-table [id=e26becdf135249f7a55ef9ddb48888ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:04.604903 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559ce6f9e5ad49409a38b3cfedd2eaea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:04.605049 2627 tablet_bootstrap.cc:654] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:04.605343 2624 tablet_bootstrap.cc:492] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28: Bootstrap starting.
I20260927 11:13:04.605602 2624 tablet_bootstrap.cc:654] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:04.606127 2675 tablet_replica.cc:330] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20: stopping tablet replica
I20260927 11:13:04.606174 2676 ts_tablet_manager.cc:1905] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:04.606227 2623 tablet_bootstrap.cc:492] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20: Bootstrap starting.
I20260927 11:13:04.606664 2623 tablet_bootstrap.cc:654] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:04.606223 2675 raft_consensus.cc:2229] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:04.607045 2675 raft_consensus.cc:2258] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:04.607993 2624 tablet_bootstrap.cc:492] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28: No bootstrap required, opened a new log
I20260927 11:13:04.608037 2676 ts_tablet_manager.cc:1918] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:13:04.608049 2624 ts_tablet_manager.cc:1397] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 11:13:04.608106 2676 log.cc:1196] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/0b754c08a5cd4289a986aeabec16ba26
I20260927 11:13:04.608201 2624 raft_consensus.cc:348] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.608280 2624 raft_consensus.cc:374] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:04.608309 2624 raft_consensus.cc:729] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731470fe975a46b59c0c291190b14a28, State: Initialized, Role: FOLLOWER
I20260927 11:13:04.608353 2624 consensus_queue.cc:260] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.608456 2624 ts_tablet_manager.cc:1428] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:04.608673 2676 ts_tablet_manager.cc:1939] T 0b754c08a5cd4289a986aeabec16ba26 P 7ebb0073746d49d99393458d0d10def7: Deleting consensus metadata
I20260927 11:13:04.608995 2107 catalog_manager.cc:4841] TS 7ebb0073746d49d99393458d0d10def7 (127.0.98.2:40711): tablet 0b754c08a5cd4289a986aeabec16ba26 (table test-table [id=be4312432d614b2a9f461661fa4e0d71]) successfully deleted
I20260927 11:13:04.609069 2627 tablet_bootstrap.cc:492] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7: No bootstrap required, opened a new log
I20260927 11:13:04.609118 2627 ts_tablet_manager.cc:1397] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 11:13:04.609258 2627 raft_consensus.cc:348] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.609342 2627 raft_consensus.cc:374] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:04.609373 2627 raft_consensus.cc:729] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ebb0073746d49d99393458d0d10def7, State: Initialized, Role: FOLLOWER
I20260927 11:13:04.609413 2627 consensus_queue.cc:260] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.609504 2627 ts_tablet_manager.cc:1428] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:04.610270 2675 ts_tablet_manager.cc:1905] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:04.611725 2675 ts_tablet_manager.cc:1918] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:13:04.611809 2675 log.cc:1196] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/0b754c08a5cd4289a986aeabec16ba26
I20260927 11:13:04.612041 2675 ts_tablet_manager.cc:1939] T 0b754c08a5cd4289a986aeabec16ba26 P 29bcf25d837b43f2b22019e510e3ac20: Deleting consensus metadata
I20260927 11:13:04.612397 2105 catalog_manager.cc:4841] TS 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3:45633): tablet 0b754c08a5cd4289a986aeabec16ba26 (table test-table [id=be4312432d614b2a9f461661fa4e0d71]) successfully deleted
I20260927 11:13:04.614346 2677 tablet_replica.cc:330] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28: stopping tablet replica
I20260927 11:13:04.614428 2623 tablet_bootstrap.cc:492] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20: No bootstrap required, opened a new log
I20260927 11:13:04.614449 2677 raft_consensus.cc:2229] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:13:04.614478 2623 ts_tablet_manager.cc:1397] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260927 11:13:04.614534 2677 raft_consensus.cc:2258] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:04.614611 2623 raft_consensus.cc:348] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.614665 2623 raft_consensus.cc:374] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:04.614692 2623 raft_consensus.cc:729] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29bcf25d837b43f2b22019e510e3ac20, State: Initialized, Role: FOLLOWER
I20260927 11:13:04.614729 2623 consensus_queue.cc:260] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.614811 2623 ts_tablet_manager.cc:1428] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:04.614856 2677 ts_tablet_manager.cc:1905] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:04.616148 2677 ts_tablet_manager.cc:1918] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:13:04.616230 2677 log.cc:1196] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/wal/wals/0b754c08a5cd4289a986aeabec16ba26
I20260927 11:13:04.616458 2677 ts_tablet_manager.cc:1939] T 0b754c08a5cd4289a986aeabec16ba26 P 731470fe975a46b59c0c291190b14a28: Deleting consensus metadata
I20260927 11:13:04.616811 2105 catalog_manager.cc:4841] TS 731470fe975a46b59c0c291190b14a28 (127.0.98.1:40171): tablet 0b754c08a5cd4289a986aeabec16ba26 (table test-table [id=be4312432d614b2a9f461661fa4e0d71]) successfully deleted
I20260927 11:13:04.687836 2630 raft_consensus.cc:482] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:04.687939 2630 raft_consensus.cc:504] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.688099 2630 leader_election.cc:288] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ebb0073746d49d99393458d0d10def7 (127.0.98.2:40711), 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3:45633)
I20260927 11:13:04.688273 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ce6f9e5ad49409a38b3cfedd2eaea" candidate_uuid: "731470fe975a46b59c0c291190b14a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bcf25d837b43f2b22019e510e3ac20" is_pre_election: true
I20260927 11:13:04.688285 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ce6f9e5ad49409a38b3cfedd2eaea" candidate_uuid: "731470fe975a46b59c0c291190b14a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ebb0073746d49d99393458d0d10def7" is_pre_election: true
I20260927 11:13:04.688385 2416 raft_consensus.cc:2454] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731470fe975a46b59c0c291190b14a28 in term 0.
I20260927 11:13:04.688385 2557 raft_consensus.cc:2454] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731470fe975a46b59c0c291190b14a28 in term 0.
I20260927 11:13:04.688905 2211 leader_election.cc:302] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731470fe975a46b59c0c291190b14a28, 7ebb0073746d49d99393458d0d10def7; no voters:
I20260927 11:13:04.689020 2630 raft_consensus.cc:2781] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:04.689064 2630 raft_consensus.cc:482] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:04.689096 2630 raft_consensus.cc:3037] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:04.689801 2630 raft_consensus.cc:504] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.689940 2630 leader_election.cc:288] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [CANDIDATE]: Term 1 election: Requested vote from peers 7ebb0073746d49d99393458d0d10def7 (127.0.98.2:40711), 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3:45633)
I20260927 11:13:04.690137 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ce6f9e5ad49409a38b3cfedd2eaea" candidate_uuid: "731470fe975a46b59c0c291190b14a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ebb0073746d49d99393458d0d10def7"
I20260927 11:13:04.690225 2416 raft_consensus.cc:3037] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:04.690208 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ce6f9e5ad49409a38b3cfedd2eaea" candidate_uuid: "731470fe975a46b59c0c291190b14a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bcf25d837b43f2b22019e510e3ac20"
I20260927 11:13:04.690274 2557 raft_consensus.cc:3037] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:04.690974 2416 raft_consensus.cc:2454] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731470fe975a46b59c0c291190b14a28 in term 1.
I20260927 11:13:04.690973 2557 raft_consensus.cc:2454] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731470fe975a46b59c0c291190b14a28 in term 1.
I20260927 11:13:04.691164 2209 leader_election.cc:302] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29bcf25d837b43f2b22019e510e3ac20, 731470fe975a46b59c0c291190b14a28; no voters:
I20260927 11:13:04.691298 2630 raft_consensus.cc:2781] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:04.691370 2630 raft_consensus.cc:686] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [term 1 LEADER]: Becoming Leader. State: Replica: 731470fe975a46b59c0c291190b14a28, State: Running, Role: LEADER
I20260927 11:13:04.691416 2630 consensus_queue.cc:237] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:04.691999 2119 catalog_manager.cc:5495] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 731470fe975a46b59c0c291190b14a28 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "731470fe975a46b59c0c291190b14a28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } health_report { overall_health: UNKNOWN } } }
I20260927 11:13:04.700935 2557 raft_consensus.cc:1261] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Refusing update from remote peer 731470fe975a46b59c0c291190b14a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:04.701038 2417 raft_consensus.cc:1261] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Refusing update from remote peer 731470fe975a46b59c0c291190b14a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:04.701112 2636 consensus_queue.cc:1035] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:04.701284 2636 consensus_queue.cc:1035] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 11:13:04.705806 2655 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 11:13:06.222486 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:06.222591 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:06.223388 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1: Sending DeleteTablet for 3 replicas of tablet 559ce6f9e5ad49409a38b3cfedd2eaea
I20260927 11:13:06.223625 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 559ce6f9e5ad49409a38b3cfedd2eaea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42148
I20260927 11:13:06.223618 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 559ce6f9e5ad49409a38b3cfedd2eaea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53742
I20260927 11:13:06.225283 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 559ce6f9e5ad49409a38b3cfedd2eaea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49424
I20260927 11:13:06.225461 2709 tablet_replica.cc:330] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20: stopping tablet replica
I20260927 11:13:06.225471 2706 tablet_replica.cc:330] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28: stopping tablet replica
I20260927 11:13:06.225541 2706 raft_consensus.cc:2229] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:13:06.225589 2709 raft_consensus.cc:2229] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:06.225636 2706 raft_consensus.cc:2258] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:06.225677 2709 raft_consensus.cc:2258] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:06.225908 2706 ts_tablet_manager.cc:1905] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:06.225908 2709 ts_tablet_manager.cc:1905] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:06.227650 2706 ts_tablet_manager.cc:1918] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 11:13:06.227702 2709 ts_tablet_manager.cc:1918] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 11:13:06.227725 2706 log.cc:1196] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/wal/wals/559ce6f9e5ad49409a38b3cfedd2eaea
I20260927 11:13:06.227803 2709 log.cc:1196] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/559ce6f9e5ad49409a38b3cfedd2eaea
I20260927 11:13:06.228021 2706 ts_tablet_manager.cc:1939] T 559ce6f9e5ad49409a38b3cfedd2eaea P 731470fe975a46b59c0c291190b14a28: Deleting consensus metadata
I20260927 11:13:06.228052 2709 ts_tablet_manager.cc:1939] T 559ce6f9e5ad49409a38b3cfedd2eaea P 29bcf25d837b43f2b22019e510e3ac20: Deleting consensus metadata
I20260927 11:13:06.228410 2105 catalog_manager.cc:4841] TS 731470fe975a46b59c0c291190b14a28 (127.0.98.1:40171): tablet 559ce6f9e5ad49409a38b3cfedd2eaea (table test-table [id=e26becdf135249f7a55ef9ddb48888ed]) successfully deleted
I20260927 11:13:06.228483 2105 catalog_manager.cc:4841] TS 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3:45633): tablet 559ce6f9e5ad49409a38b3cfedd2eaea (table test-table [id=e26becdf135249f7a55ef9ddb48888ed]) successfully deleted
I20260927 11:13:06.229493 2707 tablet_replica.cc:330] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7: stopping tablet replica
I20260927 11:13:06.229569 2707 raft_consensus.cc:2229] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:06.229612 2707 raft_consensus.cc:2258] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:06.230557 2707 ts_tablet_manager.cc:1905] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:06.232326 2707 ts_tablet_manager.cc:1918] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 11:13:06.232391 2707 log.cc:1196] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/559ce6f9e5ad49409a38b3cfedd2eaea
I20260927 11:13:06.232599 2707 ts_tablet_manager.cc:1939] T 559ce6f9e5ad49409a38b3cfedd2eaea P 7ebb0073746d49d99393458d0d10def7: Deleting consensus metadata
I20260927 11:13:06.232923 2107 catalog_manager.cc:4841] TS 7ebb0073746d49d99393458d0d10def7 (127.0.98.2:40711): tablet 559ce6f9e5ad49409a38b3cfedd2eaea (table test-table [id=e26becdf135249f7a55ef9ddb48888ed]) successfully deleted
Sep 27 11:13:06 dist-test-slave-9hq9 krb5kdc[2075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507586, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 11:13:06 dist-test-slave-9hq9 krb5kdc[2075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507586, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:06.246863 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:06.240297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41412 (local address 127.0.98.62:41885)
0927 11:13:06.240457 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:06.240460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:06.240513 (+ 53us) server_negotiation.cc:407] Connection header received
0927 11:13:06.240577 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:06.240578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:06.240587 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:06.240638 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:06.241133 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:06.241602 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:06.242305 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:06.242480 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:06.242523 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:06.244681 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:06.244682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:06.244682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:06.244688 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:06.245868 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:06.246302 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:06.246303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:06.246303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:06.246356 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:06.246502 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:06.246503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:06.246503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:06.246593 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:06.246661 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:06.246746 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 11:13:06.246804 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
I20260927 11:13:06.248167 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:13:06.248319 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:06.249524 2257 tablet_service.cc:1467] Processing CreateTablet for tablet da35988e5dac4a43ad123bade96368ca (DEFAULT_TABLE table=test-table [id=9c04ce3382a74a79919e7c612d9e0149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:06.249545 2537 tablet_service.cc:1467] Processing CreateTablet for tablet da35988e5dac4a43ad123bade96368ca (DEFAULT_TABLE table=test-table [id=9c04ce3382a74a79919e7c612d9e0149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:06.249632 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da35988e5dac4a43ad123bade96368ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:06.249632 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da35988e5dac4a43ad123bade96368ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:06.250888 2717 tablet_bootstrap.cc:492] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28: Bootstrap starting.
I20260927 11:13:06.250893 2397 tablet_service.cc:1467] Processing CreateTablet for tablet da35988e5dac4a43ad123bade96368ca (DEFAULT_TABLE table=test-table [id=9c04ce3382a74a79919e7c612d9e0149]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:06.250975 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da35988e5dac4a43ad123bade96368ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:06.251235 2717 tablet_bootstrap.cc:654] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:06.251502 2718 tablet_bootstrap.cc:492] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20: Bootstrap starting.
I20260927 11:13:06.251806 2718 tablet_bootstrap.cc:654] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:06.252180 2717 tablet_bootstrap.cc:492] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28: No bootstrap required, opened a new log
I20260927 11:13:06.252233 2717 ts_tablet_manager.cc:1397] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:06.252377 2717 raft_consensus.cc:348] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.252424 2721 tablet_bootstrap.cc:492] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7: Bootstrap starting.
I20260927 11:13:06.252460 2717 raft_consensus.cc:374] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:06.252480 2717 raft_consensus.cc:729] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731470fe975a46b59c0c291190b14a28, State: Initialized, Role: FOLLOWER
I20260927 11:13:06.252537 2717 consensus_queue.cc:260] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.252648 2717 ts_tablet_manager.cc:1428] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:06.252749 2721 tablet_bootstrap.cc:654] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:06.252858 2718 tablet_bootstrap.cc:492] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20: No bootstrap required, opened a new log
I20260927 11:13:06.252915 2718 ts_tablet_manager.cc:1397] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:06.253042 2718 raft_consensus.cc:348] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.253118 2718 raft_consensus.cc:374] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:06.253139 2718 raft_consensus.cc:729] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29bcf25d837b43f2b22019e510e3ac20, State: Initialized, Role: FOLLOWER
I20260927 11:13:06.253207 2718 consensus_queue.cc:260] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.253266 2721 tablet_bootstrap.cc:492] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7: No bootstrap required, opened a new log
I20260927 11:13:06.253304 2721 ts_tablet_manager.cc:1397] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:06.253320 2718 ts_tablet_manager.cc:1428] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:06.253434 2721 raft_consensus.cc:348] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.253496 2721 raft_consensus.cc:374] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:06.253510 2721 raft_consensus.cc:729] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ebb0073746d49d99393458d0d10def7, State: Initialized, Role: FOLLOWER
I20260927 11:13:06.253578 2721 consensus_queue.cc:260] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.253721 2721 ts_tablet_manager.cc:1428] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:06.391283 2723 raft_consensus.cc:482] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:06.391366 2723 raft_consensus.cc:504] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.391528 2723 leader_election.cc:288] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 731470fe975a46b59c0c291190b14a28 (127.0.98.1:40171), 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3:45633)
I20260927 11:13:06.391762 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da35988e5dac4a43ad123bade96368ca" candidate_uuid: "7ebb0073746d49d99393458d0d10def7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731470fe975a46b59c0c291190b14a28" is_pre_election: true
I20260927 11:13:06.391811 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da35988e5dac4a43ad123bade96368ca" candidate_uuid: "7ebb0073746d49d99393458d0d10def7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bcf25d837b43f2b22019e510e3ac20" is_pre_election: true
I20260927 11:13:06.391876 2277 raft_consensus.cc:2454] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ebb0073746d49d99393458d0d10def7 in term 0.
I20260927 11:13:06.391913 2557 raft_consensus.cc:2454] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ebb0073746d49d99393458d0d10def7 in term 0.
I20260927 11:13:06.392043 2349 leader_election.cc:302] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29bcf25d837b43f2b22019e510e3ac20, 7ebb0073746d49d99393458d0d10def7; no voters:
I20260927 11:13:06.392156 2723 raft_consensus.cc:2781] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:06.392207 2723 raft_consensus.cc:482] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:06.392238 2723 raft_consensus.cc:3037] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:06.393146 2723 raft_consensus.cc:504] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.393266 2723 leader_election.cc:288] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [CANDIDATE]: Term 1 election: Requested vote from peers 731470fe975a46b59c0c291190b14a28 (127.0.98.1:40171), 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3:45633)
I20260927 11:13:06.393416 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da35988e5dac4a43ad123bade96368ca" candidate_uuid: "7ebb0073746d49d99393458d0d10def7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bcf25d837b43f2b22019e510e3ac20"
I20260927 11:13:06.393429 2277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da35988e5dac4a43ad123bade96368ca" candidate_uuid: "7ebb0073746d49d99393458d0d10def7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731470fe975a46b59c0c291190b14a28"
I20260927 11:13:06.393483 2557 raft_consensus.cc:3037] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:06.393512 2277 raft_consensus.cc:3037] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:06.394145 2557 raft_consensus.cc:2454] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ebb0073746d49d99393458d0d10def7 in term 1.
I20260927 11:13:06.394230 2277 raft_consensus.cc:2454] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ebb0073746d49d99393458d0d10def7 in term 1.
I20260927 11:13:06.394336 2349 leader_election.cc:302] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29bcf25d837b43f2b22019e510e3ac20, 7ebb0073746d49d99393458d0d10def7; no voters:
I20260927 11:13:06.394430 2723 raft_consensus.cc:2781] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:06.394586 2723 raft_consensus.cc:686] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [term 1 LEADER]: Becoming Leader. State: Replica: 7ebb0073746d49d99393458d0d10def7, State: Running, Role: LEADER
I20260927 11:13:06.394675 2723 consensus_queue.cc:237] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.395360 2120 catalog_manager.cc:5495] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ebb0073746d49d99393458d0d10def7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7ebb0073746d49d99393458d0d10def7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } health_report { overall_health: UNKNOWN } } }
I20260927 11:13:06.434835 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:06.432258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.0.98.2:40711)
0927 11:13:06.432413 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:06.432414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:06.432421 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:13:06.432462 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:06.432462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:06.432481 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:06.432569 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:06.432701 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:06.433204 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:06.433954 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:06.434229 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:06.434313 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:06.434476 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:06.434593 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:06.434679 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:06.434721 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 11:13:06.434739 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260927 11:13:06.439114 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:06.436939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45633 (local address 127.0.98.2:55503)
0927 11:13:06.437085 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:06.437104 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:06.437149 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:06.437256 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:06.437256 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:06.437257 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:13:06.437301 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:06.437302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:06.437840 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:06.437841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:06.438476 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:06.438478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:06.438788 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:06.438789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:06.438824 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:06.438833 (+ 9us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:13:06.438957 (+ 124us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:13:06.438958 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 11:13:06.439007 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 11:13:06.439035 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
I20260927 11:13:06.439112 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:06.436986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55503 (local address 127.0.98.3:45633)
0927 11:13:06.437115 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:06.437116 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:06.437142 (+ 26us) server_negotiation.cc:407] Connection header received
0927 11:13:06.437184 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:06.437184 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:06.437193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:06.437244 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:06.437349 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:06.437789 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:06.438533 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:06.438710 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:06.438767 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:06.438890 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:06.438911 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:06.438942 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:06.439014 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 11:13:06.439030 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":91,"threads_started":1}
I20260927 11:13:06.441000 2557 raft_consensus.cc:1261] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Refusing update from remote peer 7ebb0073746d49d99393458d0d10def7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:06.441022 2277 raft_consensus.cc:1261] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28 [term 1 FOLLOWER]: Refusing update from remote peer 7ebb0073746d49d99393458d0d10def7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:06.441197 2723 consensus_queue.cc:1035] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:06.441337 2724 consensus_queue.cc:1035] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:06.448092 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:06.445854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51283 (local address 127.0.98.2:40711)
0927 11:13:06.445892 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:06.445894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:06.446012 (+ 118us) server_negotiation.cc:407] Connection header received
0927 11:13:06.446074 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:06.446074 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:06.446084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:06.446145 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:06.446241 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:06.446667 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:06.447329 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:06.447617 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:06.447695 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:06.447833 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:06.447865 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:06.447913 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:06.448028 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 11:13:06.448051 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:13:06.448094 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:06.445813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40711 (local address 127.0.98.3:51283)
0927 11:13:06.445938 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:06.445948 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:06.446001 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:06.446157 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:06.446158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:06.446160 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:13:06.446199 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:06.446200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:06.446727 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:06.446727 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:06.447263 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:06.447264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:06.447708 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:06.447709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:06.447751 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:06.447772 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:13:06.447964 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:13:06.447965 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 11:13:06.448009 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 11:13:06.448044 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
I20260927 11:13:06.454227 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:06.454315 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:06.454958 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1: Sending DeleteTablet for 3 replicas of tablet da35988e5dac4a43ad123bade96368ca
I20260927 11:13:06.455178 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet da35988e5dac4a43ad123bade96368ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53742
I20260927 11:13:06.455221 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet da35988e5dac4a43ad123bade96368ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42148
I20260927 11:13:06.455224 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet da35988e5dac4a43ad123bade96368ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49424
I20260927 11:13:06.455265 2706 tablet_replica.cc:330] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28: stopping tablet replica
I20260927 11:13:06.455295 2709 tablet_replica.cc:330] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20: stopping tablet replica
I20260927 11:13:06.455313 2707 tablet_replica.cc:330] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7: stopping tablet replica
I20260927 11:13:06.455320 2706 raft_consensus.cc:2229] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:06.455346 2709 raft_consensus.cc:2229] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:06.455351 2706 raft_consensus.cc:2258] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:06.455360 2707 raft_consensus.cc:2229] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:13:06.455380 2709 raft_consensus.cc:2258] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:06.455403 2707 raft_consensus.cc:2258] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:06.455677 2707 ts_tablet_manager.cc:1905] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:06.455793 2709 ts_tablet_manager.cc:1905] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:06.457056 2706 ts_tablet_manager.cc:1905] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:06.457381 2709 ts_tablet_manager.cc:1918] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:13:06.457394 2707 ts_tablet_manager.cc:1918] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:13:06.457432 2709 log.cc:1196] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/da35988e5dac4a43ad123bade96368ca
I20260927 11:13:06.457444 2707 log.cc:1196] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/da35988e5dac4a43ad123bade96368ca
I20260927 11:13:06.457666 2709 ts_tablet_manager.cc:1939] T da35988e5dac4a43ad123bade96368ca P 29bcf25d837b43f2b22019e510e3ac20: Deleting consensus metadata
I20260927 11:13:06.457675 2707 ts_tablet_manager.cc:1939] T da35988e5dac4a43ad123bade96368ca P 7ebb0073746d49d99393458d0d10def7: Deleting consensus metadata
I20260927 11:13:06.457944 2105 catalog_manager.cc:4841] TS 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3:45633): tablet da35988e5dac4a43ad123bade96368ca (table test-table [id=9c04ce3382a74a79919e7c612d9e0149]) successfully deleted
I20260927 11:13:06.458153 2107 catalog_manager.cc:4841] TS 7ebb0073746d49d99393458d0d10def7 (127.0.98.2:40711): tablet da35988e5dac4a43ad123bade96368ca (table test-table [id=9c04ce3382a74a79919e7c612d9e0149]) successfully deleted
I20260927 11:13:06.458629 2706 ts_tablet_manager.cc:1918] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:13:06.458675 2706 log.cc:1196] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-0/wal/wals/da35988e5dac4a43ad123bade96368ca
I20260927 11:13:06.458876 2706 ts_tablet_manager.cc:1939] T da35988e5dac4a43ad123bade96368ca P 731470fe975a46b59c0c291190b14a28: Deleting consensus metadata
I20260927 11:13:06.459136 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 11:13:06.459200 2105 catalog_manager.cc:4841] TS 731470fe975a46b59c0c291190b14a28 (127.0.98.1:40171): tablet da35988e5dac4a43ad123bade96368ca (table test-table [id=9c04ce3382a74a79919e7c612d9e0149]) successfully deleted
W20260927 11:13:06.459286 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:06.460469 2257 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea9822febe548fb9276b30efe6f6a56 (DEFAULT_TABLE table=test-table [id=35caee8facf549c0924bf8487fdaa151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:06.460485 2537 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea9822febe548fb9276b30efe6f6a56 (DEFAULT_TABLE table=test-table [id=35caee8facf549c0924bf8487fdaa151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:06.460570 2537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea9822febe548fb9276b30efe6f6a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:06.460570 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea9822febe548fb9276b30efe6f6a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:06.460819 2397 tablet_service.cc:1467] Processing CreateTablet for tablet 2ea9822febe548fb9276b30efe6f6a56 (DEFAULT_TABLE table=test-table [id=35caee8facf549c0924bf8487fdaa151]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:06.460911 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ea9822febe548fb9276b30efe6f6a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:06.461704 2717 tablet_bootstrap.cc:492] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28: Bootstrap starting.
I20260927 11:13:06.461724 2718 tablet_bootstrap.cc:492] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20: Bootstrap starting.
I20260927 11:13:06.461964 2721 tablet_bootstrap.cc:492] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7: Bootstrap starting.
I20260927 11:13:06.461998 2717 tablet_bootstrap.cc:654] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:06.461998 2718 tablet_bootstrap.cc:654] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:06.462379 2721 tablet_bootstrap.cc:654] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:06.462556 2718 tablet_bootstrap.cc:492] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20: No bootstrap required, opened a new log
I20260927 11:13:06.462620 2718 ts_tablet_manager.cc:1397] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:06.462749 2718 raft_consensus.cc:348] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.462819 2718 raft_consensus.cc:374] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:06.462847 2718 raft_consensus.cc:729] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29bcf25d837b43f2b22019e510e3ac20, State: Initialized, Role: FOLLOWER
I20260927 11:13:06.462882 2718 consensus_queue.cc:260] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.462946 2717 tablet_bootstrap.cc:492] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28: No bootstrap required, opened a new log
I20260927 11:13:06.462980 2718 ts_tablet_manager.cc:1428] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:06.462996 2717 ts_tablet_manager.cc:1397] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:06.463088 2717 raft_consensus.cc:348] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.463156 2717 raft_consensus.cc:374] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:06.463178 2717 raft_consensus.cc:729] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731470fe975a46b59c0c291190b14a28, State: Initialized, Role: FOLLOWER
I20260927 11:13:06.463218 2717 consensus_queue.cc:260] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.463303 2717 ts_tablet_manager.cc:1428] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:06.463469 2721 tablet_bootstrap.cc:492] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7: No bootstrap required, opened a new log
I20260927 11:13:06.463515 2721 ts_tablet_manager.cc:1397] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:06.463675 2721 raft_consensus.cc:348] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.463739 2721 raft_consensus.cc:374] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:06.463764 2721 raft_consensus.cc:729] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ebb0073746d49d99393458d0d10def7, State: Initialized, Role: FOLLOWER
I20260927 11:13:06.463814 2721 consensus_queue.cc:260] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.463914 2721 ts_tablet_manager.cc:1428] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:06.465487 2696 raft_consensus.cc:482] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:06.465546 2696 raft_consensus.cc:504] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.465680 2696 leader_election.cc:288] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ebb0073746d49d99393458d0d10def7 (127.0.98.2:40711), 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3:45633)
I20260927 11:13:06.465792 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea9822febe548fb9276b30efe6f6a56" candidate_uuid: "731470fe975a46b59c0c291190b14a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bcf25d837b43f2b22019e510e3ac20" is_pre_election: true
I20260927 11:13:06.465811 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea9822febe548fb9276b30efe6f6a56" candidate_uuid: "731470fe975a46b59c0c291190b14a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ebb0073746d49d99393458d0d10def7" is_pre_election: true
I20260927 11:13:06.465862 2557 raft_consensus.cc:2454] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731470fe975a46b59c0c291190b14a28 in term 0.
I20260927 11:13:06.465878 2417 raft_consensus.cc:2454] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731470fe975a46b59c0c291190b14a28 in term 0.
I20260927 11:13:06.465992 2209 leader_election.cc:302] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29bcf25d837b43f2b22019e510e3ac20, 731470fe975a46b59c0c291190b14a28; no voters:
I20260927 11:13:06.466107 2696 raft_consensus.cc:2781] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:06.466142 2696 raft_consensus.cc:482] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:06.466163 2696 raft_consensus.cc:3037] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:06.466831 2696 raft_consensus.cc:504] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.466972 2696 leader_election.cc:288] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [CANDIDATE]: Term 1 election: Requested vote from peers 7ebb0073746d49d99393458d0d10def7 (127.0.98.2:40711), 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3:45633)
I20260927 11:13:06.467059 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea9822febe548fb9276b30efe6f6a56" candidate_uuid: "731470fe975a46b59c0c291190b14a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ebb0073746d49d99393458d0d10def7"
I20260927 11:13:06.467132 2417 raft_consensus.cc:3037] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:06.467125 2557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ea9822febe548fb9276b30efe6f6a56" candidate_uuid: "731470fe975a46b59c0c291190b14a28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29bcf25d837b43f2b22019e510e3ac20"
I20260927 11:13:06.467181 2557 raft_consensus.cc:3037] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:06.467687 2417 raft_consensus.cc:2454] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731470fe975a46b59c0c291190b14a28 in term 1.
I20260927 11:13:06.467821 2557 raft_consensus.cc:2454] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731470fe975a46b59c0c291190b14a28 in term 1.
I20260927 11:13:06.467840 2211 leader_election.cc:302] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731470fe975a46b59c0c291190b14a28, 7ebb0073746d49d99393458d0d10def7; no voters:
I20260927 11:13:06.467916 2696 raft_consensus.cc:2781] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:06.467960 2696 raft_consensus.cc:686] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [term 1 LEADER]: Becoming Leader. State: Replica: 731470fe975a46b59c0c291190b14a28, State: Running, Role: LEADER
I20260927 11:13:06.468003 2696 consensus_queue.cc:237] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } }
I20260927 11:13:06.468390 2119 catalog_manager.cc:5495] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 731470fe975a46b59c0c291190b14a28 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "731470fe975a46b59c0c291190b14a28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731470fe975a46b59c0c291190b14a28" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 } health_report { overall_health: UNKNOWN } } }
I20260927 11:13:06.474822 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:06.472375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44094 (local address 127.0.98.1:40171)
0927 11:13:06.472526 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:06.472527 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:06.472534 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:13:06.472553 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:06.472554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:06.472564 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:06.472651 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:06.472751 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:06.473222 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:06.473952 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:06.474309 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:06.474379 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:06.474539 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:06.474571 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:06.474653 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:06.474714 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 11:13:06.474733 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260927 11:13:06.476728 2557 raft_consensus.cc:1261] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Refusing update from remote peer 731470fe975a46b59c0c291190b14a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:06.476728 2417 raft_consensus.cc:1261] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Refusing update from remote peer 731470fe975a46b59c0c291190b14a28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:06.476930 2701 consensus_queue.cc:1035] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ebb0073746d49d99393458d0d10def7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:06.476996 2696 consensus_queue.cc:1035] T 2ea9822febe548fb9276b30efe6f6a56 P 731470fe975a46b59c0c291190b14a28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29bcf25d837b43f2b22019e510e3ac20" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 11:13:06.480937 2688 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 11:13:07.997318 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:07.997414 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:07.998145 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f66a8e6b2f254c3eaa83deefe4f696d1: Sending DeleteTablet for 3 replicas of tablet 2ea9822febe548fb9276b30efe6f6a56
I20260927 11:13:07.998402 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea9822febe548fb9276b30efe6f6a56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42148
I20260927 11:13:07.998575 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 2192
I20260927 11:13:07.998607 2257 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea9822febe548fb9276b30efe6f6a56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53742
I20260927 11:13:07.999539 2537 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ea9822febe548fb9276b30efe6f6a56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49424
I20260927 11:13:07.999609 2767 tablet_replica.cc:330] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7: stopping tablet replica
I20260927 11:13:07.999668 2767 raft_consensus.cc:2229] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:07.999709 2767 raft_consensus.cc:2258] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:07.999938 2767 ts_tablet_manager.cc:1905] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:08.000969 2768 tablet_replica.cc:330] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20: stopping tablet replica
I20260927 11:13:08.001045 2768 raft_consensus.cc:2229] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:08.001077 2768 raft_consensus.cc:2258] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:08.001303 2768 ts_tablet_manager.cc:1905] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:08.001905 2767 ts_tablet_manager.cc:1918] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 11:13:08.001971 2767 log.cc:1196] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/2ea9822febe548fb9276b30efe6f6a56
I20260927 11:13:08.002218 2767 ts_tablet_manager.cc:1939] T 2ea9822febe548fb9276b30efe6f6a56 P 7ebb0073746d49d99393458d0d10def7: Deleting consensus metadata
I20260927 11:13:08.002533 2107 catalog_manager.cc:4841] TS 7ebb0073746d49d99393458d0d10def7 (127.0.98.2:40711): tablet 2ea9822febe548fb9276b30efe6f6a56 (table test-table [id=35caee8facf549c0924bf8487fdaa151]) successfully deleted
I20260927 11:13:08.002790 2768 ts_tablet_manager.cc:1918] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 11:13:08.002837 2768 log.cc:1196] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/2ea9822febe548fb9276b30efe6f6a56
I20260927 11:13:08.002990 2768 ts_tablet_manager.cc:1939] T 2ea9822febe548fb9276b30efe6f6a56 P 29bcf25d837b43f2b22019e510e3ac20: Deleting consensus metadata
I20260927 11:13:08.003301 2105 catalog_manager.cc:4841] TS 29bcf25d837b43f2b22019e510e3ac20 (127.0.98.3:45633): tablet 2ea9822febe548fb9276b30efe6f6a56 (table test-table [id=35caee8facf549c0924bf8487fdaa151]) successfully deleted
W20260927 11:13:08.004814 2105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40171 (error 108)
I20260927 11:13:08.004958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 2332
I20260927 11:13:08.010295 2770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:08.005144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40171 (local address 127.0.0.1:44096)
0927 11:13:08.010192 (+ 5048us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:08.010253 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5026,"thread_start_us":53,"threads_started":1}
W20260927 11:13:08.010412 2105 catalog_manager.cc:4584] TS 731470fe975a46b59c0c291190b14a28 (127.0.98.1:40171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ea9822febe548fb9276b30efe6f6a56: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40171: connect: Connection refused (error 111)
I20260927 11:13:08.011016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 2472
I20260927 11:13:08.017050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 2088
2026-09-27T11:13:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5820 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:13:08 dist-test-slave-9hq9 krb5kdc[2773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:13:08 dist-test-slave-9hq9 krb5kdc[2773](info): set up 2 sockets
Sep 27 11:13:08 dist-test-slave-9hq9 krb5kdc[2773](info): commencing operation
krb5kdc: starting...
W20260927 11:13:09.025761 392 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:13:09 dist-test-slave-9hq9 krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:13:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:13:09.134333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43567
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:09.210071 2786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:09.210246 2786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:09.210268 2786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:09.211395 2786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:09.211439 2786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:09.211453 2786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:09.211491 2786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:09.211522 2786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:09.212929 2786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:09.213148 2786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:09.213373 2786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:13:09.218271 2786 server_base.cc:1034] running on GCE node
W20260927 11:13:09.218286 2792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:09.218293 2791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:09.218508 2794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:09.218760 2786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:09.219028 2786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:09.220187 2786 hybrid_clock.cc:648] HybridClock initialized: now 1790507589220182 us; error 35 us; skew 500 ppm
Sep 27 11:13:09 dist-test-slave-9hq9 krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:09.222519 2786 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:13:09.222918 2786 webserver.cc:460] Webserver started at http://127.0.98.62:34477/ using document root <none> and password file <none>
I20260927 11:13:09.223122 2786 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:09.223165 2786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:09.223302 2786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:09.224268 2786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "e963ba0cbfc04c3497dcceaae2e5bdbe"
format_stamp: "Formatted at 2026-09-27 11:13:09 on dist-test-slave-9hq9"
server_key: "aa3f0a867dab1ade52541da1808afaa2"
server_key_iv: "33047f276a227d31d3442e428968ae07"
server_key_version: "encryptionkey@0"
I20260927 11:13:09.224606 2786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "e963ba0cbfc04c3497dcceaae2e5bdbe"
format_stamp: "Formatted at 2026-09-27 11:13:09 on dist-test-slave-9hq9"
server_key: "aa3f0a867dab1ade52541da1808afaa2"
server_key_iv: "33047f276a227d31d3442e428968ae07"
server_key_version: "encryptionkey@0"
I20260927 11:13:09.225838 2786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:13:09.226562 2801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:09.226727 2786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:09.226809 2786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "e963ba0cbfc04c3497dcceaae2e5bdbe"
format_stamp: "Formatted at 2026-09-27 11:13:09 on dist-test-slave-9hq9"
server_key: "aa3f0a867dab1ade52541da1808afaa2"
server_key_iv: "33047f276a227d31d3442e428968ae07"
server_key_version: "encryptionkey@0"
I20260927 11:13:09.226858 2786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:09.240782 2786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:09.242480 2786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:09.242581 2786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:09.245994 2786 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43567
I20260927 11:13:09.246068 2853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43567 every 8 connection(s)
I20260927 11:13:09.246521 2786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:09.247156 2854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:09.248390 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 2786
I20260927 11:13:09.248490 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:09.248663 392 external_mini_cluster.cc:1428] Setting key 801520ac578130f4787e378baaa0d088
I20260927 11:13:09.249564 2854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe: Bootstrap starting.
I20260927 11:13:09.250157 2854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:09.250434 2854 log.cc:824] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:09.251048 2854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe: No bootstrap required, opened a new log
I20260927 11:13:09.252744 2854 raft_consensus.cc:348] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e963ba0cbfc04c3497dcceaae2e5bdbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43567 } }
I20260927 11:13:09.252885 2854 raft_consensus.cc:374] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:09.252959 2854 raft_consensus.cc:729] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e963ba0cbfc04c3497dcceaae2e5bdbe, State: Initialized, Role: FOLLOWER
I20260927 11:13:09.253091 2854 consensus_queue.cc:260] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e963ba0cbfc04c3497dcceaae2e5bdbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43567 } }
I20260927 11:13:09.253181 2854 raft_consensus.cc:388] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:09.253214 2854 raft_consensus.cc:482] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:09.253268 2854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:09.253790 2854 raft_consensus.cc:504] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e963ba0cbfc04c3497dcceaae2e5bdbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43567 } }
I20260927 11:13:09.253872 2854 leader_election.cc:302] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e963ba0cbfc04c3497dcceaae2e5bdbe; no voters:
I20260927 11:13:09.254096 2854 leader_election.cc:288] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:09.254122 2859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:09.254232 2859 raft_consensus.cc:686] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [term 1 LEADER]: Becoming Leader. State: Replica: e963ba0cbfc04c3497dcceaae2e5bdbe, State: Running, Role: LEADER
I20260927 11:13:09.254324 2859 consensus_queue.cc:237] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e963ba0cbfc04c3497dcceaae2e5bdbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43567 } }
I20260927 11:13:09.254343 2854 sys_catalog.cc:564] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [sys.catalog]: configured and running, proceeding with master startup.
Sep 27 11:13:09 dist-test-slave-9hq9 krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:09.254791 2860 sys_catalog.cc:455] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e963ba0cbfc04c3497dcceaae2e5bdbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e963ba0cbfc04c3497dcceaae2e5bdbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43567 } } }
I20260927 11:13:09.254869 2860 sys_catalog.cc:458] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:09.254791 2861 sys_catalog.cc:455] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader e963ba0cbfc04c3497dcceaae2e5bdbe. Latest consensus state: current_term: 1 leader_uuid: "e963ba0cbfc04c3497dcceaae2e5bdbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e963ba0cbfc04c3497dcceaae2e5bdbe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43567 } } }
I20260927 11:13:09.255128 2861 sys_catalog.cc:458] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:09.255493 2864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:09.255964 2864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:09.257438 2864 catalog_manager.cc:1347] Generated new cluster ID: dc9bc3e5315d4d70bf289555111323ed
I20260927 11:13:09.257489 2864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:13:09.259228 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:09.249118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42454 (local address 127.0.98.62:43567)
0927 11:13:09.249292 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:09.249300 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:09.249336 (+ 36us) server_negotiation.cc:407] Connection header received
0927 11:13:09.249756 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:09.249763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:09.249805 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:09.249943 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:09.250641 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:09.251548 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:09.252295 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:09.252684 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:09.252808 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:09.255476 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:09.255480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:09.255480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:09.255492 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:09.257720 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:09.258084 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:09.258087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:09.258087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:09.258183 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:09.258385 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:09.258386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:09.258386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:09.258658 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:09.258786 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:09.258890 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 11:13:09.259026 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260927 11:13:09.269433 2864 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:09.269919 2864 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:09.277678 2864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e963ba0cbfc04c3497dcceaae2e5bdbe: Generated new TSK 0
I20260927 11:13:09.277851 2864 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:13:09.302979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43567
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:09.388146 2880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:09.388350 2880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:09.388373 2880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:09.389565 2880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:09.389606 2880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:09.389657 2880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:09.391252 2880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:09.391508 2880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:09.391732 2880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:09.397253 2885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:09.397308 2888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:09.397317 2886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:09.397450 2880 server_base.cc:1034] running on GCE node
I20260927 11:13:09.397641 2880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:09.397866 2880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:09.399003 2880 hybrid_clock.cc:648] HybridClock initialized: now 1790507589398966 us; error 52 us; skew 500 ppm
Sep 27 11:13:09 dist-test-slave-9hq9 krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:09.401317 2880 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:13:09.401719 2880 webserver.cc:460] Webserver started at http://127.0.98.1:40163/ using document root <none> and password file <none>
I20260927 11:13:09.401932 2880 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:09.401976 2880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:09.402160 2880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:09.403177 2880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "2d99f67640b242428730f32e7f6409fe"
format_stamp: "Formatted at 2026-09-27 11:13:09 on dist-test-slave-9hq9"
server_key: "1fff8c145f68e5bd09a90385deceef40"
server_key_iv: "013bdbcb9108acbd82df5dc3c95c9d2f"
server_key_version: "encryptionkey@0"
I20260927 11:13:09.403514 2880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2d99f67640b242428730f32e7f6409fe"
format_stamp: "Formatted at 2026-09-27 11:13:09 on dist-test-slave-9hq9"
server_key: "1fff8c145f68e5bd09a90385deceef40"
server_key_iv: "013bdbcb9108acbd82df5dc3c95c9d2f"
server_key_version: "encryptionkey@0"
I20260927 11:13:09.404762 2880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:13:09.405417 2895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:09.405592 2880 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:09.405682 2880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "2d99f67640b242428730f32e7f6409fe"
format_stamp: "Formatted at 2026-09-27 11:13:09 on dist-test-slave-9hq9"
server_key: "1fff8c145f68e5bd09a90385deceef40"
server_key_iv: "013bdbcb9108acbd82df5dc3c95c9d2f"
server_key_version: "encryptionkey@0"
I20260927 11:13:09.405753 2880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:09.427269 2880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:09.429251 2880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:09.429430 2880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:09.429685 2880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:09.430008 2880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:09.430040 2880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:09.430143 2880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:09.430169 2880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:09.435789 2880 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46281
I20260927 11:13:09.435889 3008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46281 every 8 connection(s)
I20260927 11:13:09.436185 2880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:13:09.438026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 2880
I20260927 11:13:09.438145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:13:09.438309 392 external_mini_cluster.cc:1428] Setting key 35d5a63e7542cf97238329aff4e4c56a
Sep 27 11:13:09 dist-test-slave-9hq9 krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 11:13:09.446122 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:09.436905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46297 (local address 127.0.98.62:43567)
0927 11:13:09.436960 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:09.436962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:09.437141 (+ 179us) server_negotiation.cc:407] Connection header received
0927 11:13:09.438134 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:09.438135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:09.438147 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:09.438208 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:09.439003 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:09.439772 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:09.440776 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:09.441126 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:09.441208 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:09.443368 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:09.443369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:09.443369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:09.443377 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:09.444738 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:09.445133 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:09.445134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:09.445134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:09.445194 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:09.445427 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:09.445428 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:09.445429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:09.445539 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:09.445611 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:09.445906 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 11:13:09.445987 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 11:13:09.446302 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:09.436891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43567 (local address 127.0.98.1:46297)
0927 11:13:09.437045 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:09.437077 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:09.437990 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:09.438286 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:09.438289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:09.438632 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:09.438902 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:09.438908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:09.439858 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:09.439858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:09.440691 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:09.440693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:09.441213 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:09.441214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:09.441309 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:09.441840 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:09.441848 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:09.443274 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:09.444841 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:09.444846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:09.444848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:09.445060 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:09.445257 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:09.445258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:09.445259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:09.445364 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:09.445603 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:09.445608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:09.445742 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 11:13:09.445856 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 11:13:09.446043 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260927 11:13:09.446763 3009 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43567
I20260927 11:13:09.446870 3009 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:09.447093 3009 heartbeater.cc:511] Master 127.0.98.62:43567 requested a full tablet report, sending...
I20260927 11:13:09.447585 2818 ts_manager.cc:194] Registered new tserver with Master: 2d99f67640b242428730f32e7f6409fe (127.0.98.1:46281)
I20260927 11:13:09.448137 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:13:09.460901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43567
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:09.537202 3014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:09.537410 3014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:09.537446 3014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:09.538547 3014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:09.538607 3014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:09.538650 3014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:09.540053 3014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:09.540282 3014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:09.540503 3014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:09.545110 3020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:09.545148 3019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:09.545153 3022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:09.545331 3014 server_base.cc:1034] running on GCE node
I20260927 11:13:09.545542 3014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:09.545763 3014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:09.546914 3014 hybrid_clock.cc:648] HybridClock initialized: now 1790507589546911 us; error 39 us; skew 500 ppm
Sep 27 11:13:09 dist-test-slave-9hq9 krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:09.552525 3014 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:13:09.552870 3014 webserver.cc:460] Webserver started at http://127.0.98.2:38679/ using document root <none> and password file <none>
I20260927 11:13:09.553037 3014 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:09.553076 3014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:09.553155 3014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:09.554107 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "18429190d94f40f78fa0f453d7a885c1"
format_stamp: "Formatted at 2026-09-27 11:13:09 on dist-test-slave-9hq9"
server_key: "7706494132b6c6c02070983075893a76"
server_key_iv: "97a80be11b2d80066bea70b9cd7ff0c7"
server_key_version: "encryptionkey@0"
I20260927 11:13:09.554490 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "18429190d94f40f78fa0f453d7a885c1"
format_stamp: "Formatted at 2026-09-27 11:13:09 on dist-test-slave-9hq9"
server_key: "7706494132b6c6c02070983075893a76"
server_key_iv: "97a80be11b2d80066bea70b9cd7ff0c7"
server_key_version: "encryptionkey@0"
I20260927 11:13:09.555757 3014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:09.556416 3029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:09.556560 3014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:09.556645 3014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "18429190d94f40f78fa0f453d7a885c1"
format_stamp: "Formatted at 2026-09-27 11:13:09 on dist-test-slave-9hq9"
server_key: "7706494132b6c6c02070983075893a76"
server_key_iv: "97a80be11b2d80066bea70b9cd7ff0c7"
server_key_version: "encryptionkey@0"
I20260927 11:13:09.556694 3014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:09.572476 3014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:09.574602 3014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:09.574772 3014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:09.574942 3014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:09.575197 3014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:09.575227 3014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:09.575245 3014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:09.575258 3014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:09.580394 3014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42397
I20260927 11:13:09.580467 3142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42397 every 8 connection(s)
I20260927 11:13:09.580739 3014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:13:09.584967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 3014
I20260927 11:13:09.585065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:13:09.585273 392 external_mini_cluster.cc:1428] Setting key 5d2c636b189cecea0a5ab21a5fa3105c
Sep 27 11:13:09 dist-test-slave-9hq9 krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:09.589864 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:09.581432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36025 (local address 127.0.98.62:43567)
0927 11:13:09.581511 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:09.581513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:09.581730 (+ 217us) server_negotiation.cc:407] Connection header received
0927 11:13:09.582668 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:09.582668 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:09.582678 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:09.582733 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:09.583256 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:09.583731 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:09.584625 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:09.584816 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:09.584865 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:09.587108 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:09.587109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:09.587109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:09.587117 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:09.588533 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:09.588945 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:09.588946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:09.588947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:09.589002 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:09.589235 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:09.589236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:09.589237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:09.589350 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:09.589441 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:09.589701 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 11:13:09.589778 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 11:13:09.590097 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:09.581423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43567 (local address 127.0.98.2:36025)
0927 11:13:09.581602 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:09.581650 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:09.582571 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:09.582782 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:09.582785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:09.583010 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:09.583180 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:09.583184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:09.583803 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:09.583803 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:09.584556 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:09.584558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:09.584898 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:09.584899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:09.585044 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:09.585532 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:09.585542 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:09.587013 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:09.588640 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:09.588646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:09.588649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:09.588857 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:09.589066 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:09.589067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:09.589067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:09.589171 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:09.589432 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:09.589437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:09.589581 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 11:13:09.589681 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 11:13:09.589828 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
I20260927 11:13:09.590519 3143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43567
I20260927 11:13:09.590600 3143 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:09.590771 3143 heartbeater.cc:511] Master 127.0.98.62:43567 requested a full tablet report, sending...
I20260927 11:13:09.591113 2818 ts_manager.cc:194] Registered new tserver with Master: 18429190d94f40f78fa0f453d7a885c1 (127.0.98.2:42397)
I20260927 11:13:09.591495 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36025
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:13:09.609907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43567
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:09.687606 3148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:09.687803 3148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:09.687824 3148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:09.688947 3148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:09.688989 3148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:09.689044 3148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:09.690553 3148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:09.690778 3148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:09.690995 3148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:09.695832 3154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:09.695838 3156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:09.695933 3153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:09.696009 3148 server_base.cc:1034] running on GCE node
I20260927 11:13:09.696305 3148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:09.696527 3148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:09.697661 3148 hybrid_clock.cc:648] HybridClock initialized: now 1790507589697655 us; error 31 us; skew 500 ppm
Sep 27 11:13:09 dist-test-slave-9hq9 krb5kdc[2773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:09.700405 3148 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:13:09.700758 3148 webserver.cc:460] Webserver started at http://127.0.98.3:35373/ using document root <none> and password file <none>
I20260927 11:13:09.700927 3148 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:09.700966 3148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:09.701049 3148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:09.701985 3148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "144e14147b2f4db8b39bfba0461bbc5a"
format_stamp: "Formatted at 2026-09-27 11:13:09 on dist-test-slave-9hq9"
server_key: "390587faaa8c6675ea378d48496e1bc1"
server_key_iv: "92ef51f03288a820b92497260d59fa1f"
server_key_version: "encryptionkey@0"
I20260927 11:13:09.702342 3148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "144e14147b2f4db8b39bfba0461bbc5a"
format_stamp: "Formatted at 2026-09-27 11:13:09 on dist-test-slave-9hq9"
server_key: "390587faaa8c6675ea378d48496e1bc1"
server_key_iv: "92ef51f03288a820b92497260d59fa1f"
server_key_version: "encryptionkey@0"
I20260927 11:13:09.703553 3148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:09.704277 3163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:09.704444 3148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:09.704510 3148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "144e14147b2f4db8b39bfba0461bbc5a"
format_stamp: "Formatted at 2026-09-27 11:13:09 on dist-test-slave-9hq9"
server_key: "390587faaa8c6675ea378d48496e1bc1"
server_key_iv: "92ef51f03288a820b92497260d59fa1f"
server_key_version: "encryptionkey@0"
I20260927 11:13:09.704589 3148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:09.716136 3148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:09.717732 3148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:09.717834 3148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:09.717991 3148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:09.718302 3148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:09.718334 3148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:09.718354 3148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:09.718374 3148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:09.723671 3148 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44647
I20260927 11:13:09.723747 3276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44647 every 8 connection(s)
I20260927 11:13:09.724023 3148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 11:13:09 dist-test-slave-9hq9 krb5kdc[2773](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:09.733898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 3148
I20260927 11:13:09.733994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:13:09.734207 392 external_mini_cluster.cc:1428] Setting key 132fadd080a64c5fc01da762634431eb
I20260927 11:13:09.734213 3279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:09.724757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43567 (local address 127.0.98.3:45987)
0927 11:13:09.724903 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:09.724940 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:09.726290 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:09.726525 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:09.726529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:09.726923 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:09.727171 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:09.727177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:09.727828 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:09.727830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:09.729469 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:09.729472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:09.729797 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:09.729798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:09.729944 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:09.730465 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:09.730474 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:09.731863 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:09.733099 (+ 1236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:09.733104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:09.733105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:09.733305 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:09.733462 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:09.733462 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:09.733462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:09.733540 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:09.733700 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:09.733703 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:09.733794 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 11:13:09.733873 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 11:13:09.734014 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260927 11:13:09.734216 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:09.724789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45987 (local address 127.0.98.62:43567)
0927 11:13:09.724845 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:09.724847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:09.725038 (+ 191us) server_negotiation.cc:407] Connection header received
0927 11:13:09.726390 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:09.726391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:09.726401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:09.726451 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:09.727256 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:09.727730 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:09.729543 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:09.729721 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:09.729775 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:09.731937 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:09.731938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:09.731938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:09.731944 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:09.733036 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:09.733375 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:09.733375 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:09.733376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:09.733422 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:09.733582 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:09.733582 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:09.733583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:09.733666 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:09.733710 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:09.733936 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 11:13:09.734023 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 11:13:09.734937 3277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43567
I20260927 11:13:09.735025 3277 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:09.735195 3277 heartbeater.cc:511] Master 127.0.98.62:43567 requested a full tablet report, sending...
I20260927 11:13:09.735517 2818 ts_manager.cc:194] Registered new tserver with Master: 144e14147b2f4db8b39bfba0461bbc5a (127.0.98.3:44647)
I20260927 11:13:09.735893 2818 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45987
I20260927 11:13:09.745949 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 11:13:09.756553 2857 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:09.755647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.0.98.62:43567)
0927 11:13:09.755726 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:09.755728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:09.755830 (+ 102us) server_negotiation.cc:407] Connection header received
0927 11:13:09.755881 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:09.755882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:09.755891 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:09.755941 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:09.756466 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42456: BlockingRecv error: recv got EOF from 127.0.0.1:42456 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 11:13:09.757439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 2880
I20260927 11:13:09.761978 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 3014
I20260927 11:13:09.766145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 3148
I20260927 11:13:09.770190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 2786
2026-09-27T11:13:09Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1752 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:13:11 dist-test-slave-9hq9 krb5kdc[3289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:13:11 dist-test-slave-9hq9 krb5kdc[3289](info): set up 2 sockets
Sep 27 11:13:11 dist-test-slave-9hq9 krb5kdc[3289](info): commencing operation
krb5kdc: starting...
W20260927 11:13:12.193960 392 mini_kdc.cc:121] Time spent starting KDC: real 2.404s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:13:12 dist-test-slave-9hq9 krb5kdc[3289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:13:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:13:12.298911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39411
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:12.374612 3302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:12.374782 3302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:12.374804 3302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:12.375874 3302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:12.375912 3302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:12.375926 3302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:12.375974 3302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:12.375994 3302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:12.377413 3302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:12.377611 3302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:12.377816 3302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:12.382089 3307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:12.382143 3308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:12.382210 3310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:12.382395 3302 server_base.cc:1034] running on GCE node
I20260927 11:13:12.382611 3302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:12.382867 3302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:12.384017 3302 hybrid_clock.cc:648] HybridClock initialized: now 1790507592384018 us; error 44 us; skew 500 ppm
Sep 27 11:13:12 dist-test-slave-9hq9 krb5kdc[3289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:12.386214 3302 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:13:12.386675 3302 webserver.cc:460] Webserver started at http://127.0.98.62:46313/ using document root <none> and password file <none>
I20260927 11:13:12.386905 3302 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:12.386969 3302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:12.387096 3302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:12.388039 3302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "fd641440b143428faab12ef7be67d99c"
format_stamp: "Formatted at 2026-09-27 11:13:12 on dist-test-slave-9hq9"
server_key: "7f7c0e839f12918aed2c15a6c30b3384"
server_key_iv: "bdc1700eac3207d19df29af88d88e5e9"
server_key_version: "encryptionkey@0"
I20260927 11:13:12.388371 3302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "fd641440b143428faab12ef7be67d99c"
format_stamp: "Formatted at 2026-09-27 11:13:12 on dist-test-slave-9hq9"
server_key: "7f7c0e839f12918aed2c15a6c30b3384"
server_key_iv: "bdc1700eac3207d19df29af88d88e5e9"
server_key_version: "encryptionkey@0"
I20260927 11:13:12.389560 3302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:13:12.390273 3317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:12.390535 3302 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:12.390620 3302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "fd641440b143428faab12ef7be67d99c"
format_stamp: "Formatted at 2026-09-27 11:13:12 on dist-test-slave-9hq9"
server_key: "7f7c0e839f12918aed2c15a6c30b3384"
server_key_iv: "bdc1700eac3207d19df29af88d88e5e9"
server_key_version: "encryptionkey@0"
I20260927 11:13:12.390699 3302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:12.412122 3302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:12.413744 3302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:12.413894 3302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:12.417510 3302 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39411
I20260927 11:13:12.417541 3369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39411 every 8 connection(s)
I20260927 11:13:12.417838 3302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:12.418555 3370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:12.420439 3370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c: Bootstrap starting.
I20260927 11:13:12.420908 3370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:12.421128 3370 log.cc:824] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:12.421653 3370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c: No bootstrap required, opened a new log
I20260927 11:13:12.422683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 3302
I20260927 11:13:12.422785 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:12.422964 392 external_mini_cluster.cc:1428] Setting key 555624a9b538bba0c7063f8ce92119ae
I20260927 11:13:12.424073 3370 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd641440b143428faab12ef7be67d99c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39411 } }
I20260927 11:13:12.424239 3370 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:12.424274 3370 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd641440b143428faab12ef7be67d99c, State: Initialized, Role: FOLLOWER
I20260927 11:13:12.424427 3370 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd641440b143428faab12ef7be67d99c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39411 } }
I20260927 11:13:12.424513 3370 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:12.424546 3370 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:12.424588 3370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:12.425176 3370 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd641440b143428faab12ef7be67d99c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39411 } }
I20260927 11:13:12.425256 3370 leader_election.cc:302] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd641440b143428faab12ef7be67d99c; no voters:
I20260927 11:13:12.425431 3370 leader_election.cc:288] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:12.425453 3375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:12.425549 3375 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [term 1 LEADER]: Becoming Leader. State: Replica: fd641440b143428faab12ef7be67d99c, State: Running, Role: LEADER
I20260927 11:13:12.425634 3375 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd641440b143428faab12ef7be67d99c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39411 } }
I20260927 11:13:12.425657 3370 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:12.425945 3375 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd641440b143428faab12ef7be67d99c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd641440b143428faab12ef7be67d99c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39411 } } }
I20260927 11:13:12.426136 3375 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [sys.catalog]: This master's current role is: LEADER
W20260927 11:13:12.427358 3389 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:13:12.427445 3389 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:13:12.426311 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd641440b143428faab12ef7be67d99c. Latest consensus state: current_term: 1 leader_uuid: "fd641440b143428faab12ef7be67d99c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd641440b143428faab12ef7be67d99c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39411 } } }
I20260927 11:13:12.427799 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:12.427711 3391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:12.427994 3391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:12.429492 3391 catalog_manager.cc:1347] Generated new cluster ID: d1496aaf08ed470683f143c97e743eb3
I20260927 11:13:12.429543 3391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:13:12 dist-test-slave-9hq9 krb5kdc[3289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:12.433579 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:12.423413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.0.98.62:39411)
0927 11:13:12.423586 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:12.423592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:12.423614 (+ 22us) server_negotiation.cc:407] Connection header received
0927 11:13:12.424070 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:12.424075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:12.424116 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:12.424230 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:12.424962 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:12.426184 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:12.427298 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:12.427918 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:12.428010 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:12.430485 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:12.430489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:12.430489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:12.430499 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:12.432135 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:12.432502 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:12.432505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:12.432505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:12.432603 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:12.432818 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:12.432819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:12.432819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:12.433033 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:12.433226 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:12.433302 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 11:13:12.433429 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260927 11:13:12.434589 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:13:12.452442 3391 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:12.452756 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:12.449215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.0.98.62:39411)
0927 11:13:12.449254 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:12.449256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:12.449384 (+ 128us) server_negotiation.cc:407] Connection header received
0927 11:13:12.449455 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:12.449456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:12.449467 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:12.449537 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:12.449889 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:12.450430 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:12.451127 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:12.451332 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:12.451413 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:12.452231 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:12.452232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:12.452232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:12.452464 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:12.452479 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:12.452486 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:12.452541 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:12.452640 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 11:13:12.452662 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:13:12.453114 3391 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:12.460567 3391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd641440b143428faab12ef7be67d99c: Generated new TSK 0
I20260927 11:13:12.460745 3391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:13:12.464811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 3302
2026-09-27T11:13:12Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2692 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:13:12 dist-test-slave-9hq9 krb5kdc[3403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:13:12 dist-test-slave-9hq9 krb5kdc[3403](info): set up 2 sockets
Sep 27 11:13:12 dist-test-slave-9hq9 krb5kdc[3403](info): commencing operation
krb5kdc: starting...
W20260927 11:13:13.473941 392 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:13:13 dist-test-slave-9hq9 krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:13:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:13:13.594666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37081
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:13.670600 3416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:13.670779 3416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:13.670800 3416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:13.671924 3416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:13.671970 3416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:13.671984 3416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:13.671996 3416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:13.672008 3416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:13.673417 3416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:13.673643 3416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:13.673858 3416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:13.678643 3422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:13.678812 3416 server_base.cc:1034] running on GCE node
W20260927 11:13:13.678632 3424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:13.678632 3421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:13.679124 3416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:13.679442 3416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:13.680598 3416 hybrid_clock.cc:648] HybridClock initialized: now 1790507593680586 us; error 28 us; skew 500 ppm
Sep 27 11:13:13 dist-test-slave-9hq9 krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:13.682838 3416 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:13:13.683274 3416 webserver.cc:460] Webserver started at http://127.0.98.62:43497/ using document root <none> and password file <none>
I20260927 11:13:13.683487 3416 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:13.683557 3416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:13.683700 3416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:13.684656 3416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "03c3e8ac55644f5eb1ffc42e6bb24c0c"
format_stamp: "Formatted at 2026-09-27 11:13:13 on dist-test-slave-9hq9"
server_key: "0215b554d6cd64b76c0073e5e464696b"
server_key_iv: "2d54d0602b9004fb04bbb05592ef9406"
server_key_version: "encryptionkey@0"
I20260927 11:13:13.684995 3416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "03c3e8ac55644f5eb1ffc42e6bb24c0c"
format_stamp: "Formatted at 2026-09-27 11:13:13 on dist-test-slave-9hq9"
server_key: "0215b554d6cd64b76c0073e5e464696b"
server_key_iv: "2d54d0602b9004fb04bbb05592ef9406"
server_key_version: "encryptionkey@0"
I20260927 11:13:13.686272 3416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:13:13.686939 3431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:13.687081 3416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:13.687181 3416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "03c3e8ac55644f5eb1ffc42e6bb24c0c"
format_stamp: "Formatted at 2026-09-27 11:13:13 on dist-test-slave-9hq9"
server_key: "0215b554d6cd64b76c0073e5e464696b"
server_key_iv: "2d54d0602b9004fb04bbb05592ef9406"
server_key_version: "encryptionkey@0"
I20260927 11:13:13.687251 3416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:13.712170 3416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:13.714443 3416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:13.714589 3416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:13.718250 3416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37081
I20260927 11:13:13.718317 3483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37081 every 8 connection(s)
I20260927 11:13:13.718641 3416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:13.719066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 3416
I20260927 11:13:13.719144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:13.719332 392 external_mini_cluster.cc:1428] Setting key 283f9f7efce74e9d462a59cfce4e4341
I20260927 11:13:13.719751 3484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:13.722900 3484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c: Bootstrap starting.
I20260927 11:13:13.723531 3484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:13.723819 3484 log.cc:824] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:13.724486 3484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c: No bootstrap required, opened a new log
Sep 27 11:13:13 dist-test-slave-9hq9 krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:13.726336 3484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c3e8ac55644f5eb1ffc42e6bb24c0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37081 } }
I20260927 11:13:13.726533 3484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:13.726575 3484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03c3e8ac55644f5eb1ffc42e6bb24c0c, State: Initialized, Role: FOLLOWER
I20260927 11:13:13.726709 3484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c3e8ac55644f5eb1ffc42e6bb24c0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37081 } }
I20260927 11:13:13.726830 3484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:13.726864 3484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:13.726920 3484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:13.727617 3484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c3e8ac55644f5eb1ffc42e6bb24c0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37081 } }
I20260927 11:13:13.727718 3484 leader_election.cc:302] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03c3e8ac55644f5eb1ffc42e6bb24c0c; no voters:
I20260927 11:13:13.727867 3484 leader_election.cc:288] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:13.727910 3489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:13.728087 3489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [term 1 LEADER]: Becoming Leader. State: Replica: 03c3e8ac55644f5eb1ffc42e6bb24c0c, State: Running, Role: LEADER
I20260927 11:13:13.728138 3484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:13.729409 3489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c3e8ac55644f5eb1ffc42e6bb24c0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37081 } }
I20260927 11:13:13.729933 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:13.719810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55154 (local address 127.0.98.62:37081)
0927 11:13:13.719988 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:13.719997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:13.720041 (+ 44us) server_negotiation.cc:407] Connection header received
0927 11:13:13.720535 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:13.720543 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:13.720604 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:13.720750 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:13.721281 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:13.722635 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:13.723376 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:13.723999 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:13.724102 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:13.726305 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:13.726309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:13.726310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:13.726321 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:13.728270 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:13.728690 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:13.728692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:13.728693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:13.728789 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:13.728983 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:13.728984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:13.728984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:13.729317 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:13.729471 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:13.729583 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 11:13:13.729715 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260927 11:13:13.730015 3490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03c3e8ac55644f5eb1ffc42e6bb24c0c. Latest consensus state: current_term: 1 leader_uuid: "03c3e8ac55644f5eb1ffc42e6bb24c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c3e8ac55644f5eb1ffc42e6bb24c0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37081 } } }
I20260927 11:13:13.730163 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03c3e8ac55644f5eb1ffc42e6bb24c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03c3e8ac55644f5eb1ffc42e6bb24c0c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37081 } } }
I20260927 11:13:13.730688 3490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:13.730690 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c [sys.catalog]: This master's current role is: LEADER
W20260927 11:13:13.731029 3503 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:13:13.731091 3503 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:13:13.731204 3506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:13.731349 3506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:13.732723 3506 catalog_manager.cc:1347] Generated new cluster ID: 47667341e392489496794db56d8085f4
I20260927 11:13:13.732779 3506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:13:13.744611 3506 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:13.745126 3506 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:13.750746 3506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03c3e8ac55644f5eb1ffc42e6bb24c0c: Generated new TSK 0
I20260927 11:13:13.750916 3506 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:13:13.782434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37081
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:13.860443 3510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:13.860658 3510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:13.860697 3510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:13.861865 3510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:13.861927 3510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:13.861970 3510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:13.863379 3510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:13.863633 3510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:13.863837 3510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:13.868562 3516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:13.868603 3518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:13.868623 3515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:13.868750 3510 server_base.cc:1034] running on GCE node
I20260927 11:13:13.868963 3510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:13.869181 3510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:13.870312 3510 hybrid_clock.cc:648] HybridClock initialized: now 1790507593870292 us; error 35 us; skew 500 ppm
Sep 27 11:13:13 dist-test-slave-9hq9 krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:13.872704 3510 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:13:13.873052 3510 webserver.cc:460] Webserver started at http://127.0.98.1:37197/ using document root <none> and password file <none>
I20260927 11:13:13.873224 3510 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:13.873262 3510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:13.873343 3510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:13.874300 3510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "9f9e83c4d23b478394dd9411abc5025e"
format_stamp: "Formatted at 2026-09-27 11:13:13 on dist-test-slave-9hq9"
server_key: "b725e56dd6b31c5e63d6dd67f81b96be"
server_key_iv: "2b8e6d73a7ac35cc5d2337fcbfb7e513"
server_key_version: "encryptionkey@0"
I20260927 11:13:13.874596 3510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9f9e83c4d23b478394dd9411abc5025e"
format_stamp: "Formatted at 2026-09-27 11:13:13 on dist-test-slave-9hq9"
server_key: "b725e56dd6b31c5e63d6dd67f81b96be"
server_key_iv: "2b8e6d73a7ac35cc5d2337fcbfb7e513"
server_key_version: "encryptionkey@0"
I20260927 11:13:13.875741 3510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:13.876377 3525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:13.876571 3510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:13.876675 3510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "9f9e83c4d23b478394dd9411abc5025e"
format_stamp: "Formatted at 2026-09-27 11:13:13 on dist-test-slave-9hq9"
server_key: "b725e56dd6b31c5e63d6dd67f81b96be"
server_key_iv: "2b8e6d73a7ac35cc5d2337fcbfb7e513"
server_key_version: "encryptionkey@0"
I20260927 11:13:13.876757 3510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:13.911471 3510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:13.913643 3510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:13.913800 3510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:13.914016 3510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:13.914397 3510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:13.914451 3510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:13.914490 3510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:13.914516 3510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:13.920127 3510 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35131
I20260927 11:13:13.920173 3638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35131 every 8 connection(s)
I20260927 11:13:13.920470 3510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:13:13 dist-test-slave-9hq9 krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:13.926872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 3510
I20260927 11:13:13.926946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:13:13.927125 392 external_mini_cluster.cc:1428] Setting key 9d0fcf47fc99367449fcf74dd231bc94
I20260927 11:13:13.929454 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:13.921117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34185 (local address 127.0.98.62:37081)
0927 11:13:13.921166 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:13.921169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:13.921419 (+ 250us) server_negotiation.cc:407] Connection header received
0927 11:13:13.921852 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:13.921853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:13.921862 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:13.921912 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:13.922805 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:13.923283 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:13.924240 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:13.924491 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:13.924541 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:13.926401 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:13.926402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:13.926403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:13.926408 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:13.927924 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:13.928426 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:13.928427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:13.928427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:13.928489 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:13.928746 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:13.928747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:13.928747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:13.928836 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:13.928886 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:13.929262 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 11:13:13.929347 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:13:13.929597 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:13.921095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37081 (local address 127.0.98.1:34185)
0927 11:13:13.921250 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:13.921317 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:13.921772 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:13.921958 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:13.921961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:13.922560 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:13.922729 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:13.922734 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:13.923356 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:13.923357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:13.924170 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:13.924172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:13.924559 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:13.924560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:13.924651 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:13.925045 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:13.925053 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:13.926323 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:13.928028 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:13.928034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:13.928036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:13.928323 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:13.928539 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:13.928540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:13.928540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:13.928691 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:13.928899 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:13.928904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:13.929110 (+ 206us) client_negotiation.cc:769] Sending connection context
0927 11:13:13.929230 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 11:13:13.929345 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260927 11:13:13.930042 3639 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37081
I20260927 11:13:13.930142 3639 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:13.930302 3639 heartbeater.cc:511] Master 127.0.98.62:37081 requested a full tablet report, sending...
I20260927 11:13:13.930759 3448 ts_manager.cc:194] Registered new tserver with Master: 9f9e83c4d23b478394dd9411abc5025e (127.0.98.1:35131)
I20260927 11:13:13.931453 3448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34185
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:13:13.950708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37081
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:14.027335 3644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:14.027525 3644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:14.027546 3644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:14.028780 3644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:14.028818 3644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:14.028843 3644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:14.030334 3644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:14.030548 3644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:14.030745 3644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:13:14.035477 3644 server_base.cc:1034] running on GCE node
W20260927 11:13:14.035584 3649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:14.035486 3650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:14.035591 3652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:14.035917 3644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:14.036171 3644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:14.037310 3644 hybrid_clock.cc:648] HybridClock initialized: now 1790507594037292 us; error 30 us; skew 500 ppm
Sep 27 11:13:14 dist-test-slave-9hq9 krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:14.039633 3644 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:13:14.040058 3644 webserver.cc:460] Webserver started at http://127.0.98.2:34215/ using document root <none> and password file <none>
I20260927 11:13:14.040259 3644 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:14.040329 3644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:14.040468 3644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:14.041379 3644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "72b0ffe7562f4c5ca37af9d0feb91eb2"
format_stamp: "Formatted at 2026-09-27 11:13:14 on dist-test-slave-9hq9"
server_key: "e552be2bd636ccbbab136179f6b68ffe"
server_key_iv: "e48206179962f6df07044334e8ac432f"
server_key_version: "encryptionkey@0"
I20260927 11:13:14.041702 3644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "72b0ffe7562f4c5ca37af9d0feb91eb2"
format_stamp: "Formatted at 2026-09-27 11:13:14 on dist-test-slave-9hq9"
server_key: "e552be2bd636ccbbab136179f6b68ffe"
server_key_iv: "e48206179962f6df07044334e8ac432f"
server_key_version: "encryptionkey@0"
I20260927 11:13:14.043017 3644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:13:14.043658 3659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:14.043812 3644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:14.043901 3644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "72b0ffe7562f4c5ca37af9d0feb91eb2"
format_stamp: "Formatted at 2026-09-27 11:13:14 on dist-test-slave-9hq9"
server_key: "e552be2bd636ccbbab136179f6b68ffe"
server_key_iv: "e48206179962f6df07044334e8ac432f"
server_key_version: "encryptionkey@0"
I20260927 11:13:14.043982 3644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:14.058705 3644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:14.060670 3644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:14.060803 3644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:14.060999 3644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:14.061259 3644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:14.061313 3644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:14.061348 3644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:14.061375 3644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:14.067221 3644 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39293
I20260927 11:13:14.067253 3772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39293 every 8 connection(s)
I20260927 11:13:14.067540 3644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:13:14 dist-test-slave-9hq9 krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:14.074679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 3644
I20260927 11:13:14.074782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:13:14.074967 392 external_mini_cluster.cc:1428] Setting key cf789401fc1ce69181394b53dc9ca5d4
I20260927 11:13:14.075862 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:14.068159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50075 (local address 127.0.98.62:37081)
0927 11:13:14.068217 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:14.068219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:14.068427 (+ 208us) server_negotiation.cc:407] Connection header received
0927 11:13:14.068825 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:14.068826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:14.068836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:14.068885 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:14.069657 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:14.070196 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:14.071051 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:14.071218 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:14.071264 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:14.073016 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:14.073018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:14.073018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:14.073024 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:14.074441 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:14.074899 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:14.074900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:14.074900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:14.074950 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:14.075291 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:14.075292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:14.075292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:14.075404 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:14.075461 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:14.075731 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 11:13:14.075803 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 11:13:14.076166 3775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:14.068141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37081 (local address 127.0.98.2:50075)
0927 11:13:14.068294 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:14.068322 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:14.068751 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:14.068943 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:14.068946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:14.069434 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:14.069596 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:14.069599 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:14.070263 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:14.070263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:14.070978 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:14.070980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:14.071279 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:14.071280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:14.071364 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:14.071715 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:14.071722 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:14.072940 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:14.074503 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:14.074508 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:14.074523 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:14.074817 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:14.075027 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:14.075028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:14.075028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:14.075240 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:14.075480 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:14.075485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:14.075602 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 11:13:14.075717 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 11:13:14.075889 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260927 11:13:14.076599 3773 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37081
I20260927 11:13:14.076685 3773 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:14.076853 3773 heartbeater.cc:511] Master 127.0.98.62:37081 requested a full tablet report, sending...
I20260927 11:13:14.077219 3448 ts_manager.cc:194] Registered new tserver with Master: 72b0ffe7562f4c5ca37af9d0feb91eb2 (127.0.98.2:39293)
I20260927 11:13:14.077718 3448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50075
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:13:14.096863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37081
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:14.176775 3778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:14.176960 3778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:14.176982 3778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:14.178032 3778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:14.178128 3778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:14.178159 3778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:14.179690 3778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37081
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:14.179853 3778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:14.180006 3778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:14.184558 3784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:14.184553 3783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:14.184636 3778 server_base.cc:1034] running on GCE node
W20260927 11:13:14.184726 3786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:14.184973 3778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:14.185212 3778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:14.186370 3778 hybrid_clock.cc:648] HybridClock initialized: now 1790507594186352 us; error 37 us; skew 500 ppm
Sep 27 11:13:14 dist-test-slave-9hq9 krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:14.188589 3778 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:13:14.188952 3778 webserver.cc:460] Webserver started at http://127.0.98.3:45263/ using document root <none> and password file <none>
I20260927 11:13:14.189217 3778 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:14.189277 3778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:14.189399 3778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:14.190478 3778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "f64cb34c8bcd4634bdc61d2c35502e6c"
format_stamp: "Formatted at 2026-09-27 11:13:14 on dist-test-slave-9hq9"
server_key: "bfc6a22897b3595cdbb585b74907f072"
server_key_iv: "c98bce8ba16710507908bfab0b9f0979"
server_key_version: "encryptionkey@0"
I20260927 11:13:14.190828 3778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f64cb34c8bcd4634bdc61d2c35502e6c"
format_stamp: "Formatted at 2026-09-27 11:13:14 on dist-test-slave-9hq9"
server_key: "bfc6a22897b3595cdbb585b74907f072"
server_key_iv: "c98bce8ba16710507908bfab0b9f0979"
server_key_version: "encryptionkey@0"
I20260927 11:13:14.192019 3778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:13:14.192663 3793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:14.192822 3778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:14.192916 3778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "f64cb34c8bcd4634bdc61d2c35502e6c"
format_stamp: "Formatted at 2026-09-27 11:13:14 on dist-test-slave-9hq9"
server_key: "bfc6a22897b3595cdbb585b74907f072"
server_key_iv: "c98bce8ba16710507908bfab0b9f0979"
server_key_version: "encryptionkey@0"
I20260927 11:13:14.193022 3778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:14.221896 3778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:14.223623 3778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:14.223771 3778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:14.223975 3778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:14.224244 3778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:14.224295 3778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:14.224339 3778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:14.224367 3778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:14.230320 3778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33817
I20260927 11:13:14.230371 3906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33817 every 8 connection(s)
I20260927 11:13:14.230636 3778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:13:14.231348 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 3778
I20260927 11:13:14.231431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:13:14.231601 392 external_mini_cluster.cc:1428] Setting key 95ec8802bd997376f19faf9d632dda58
Sep 27 11:13:14 dist-test-slave-9hq9 krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:14.239322 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:14.231259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36415 (local address 127.0.98.62:37081)
0927 11:13:14.231306 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:14.231307 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:14.231536 (+ 229us) server_negotiation.cc:407] Connection header received
0927 11:13:14.232122 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:14.232123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:14.232136 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:14.232183 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:14.233037 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:14.233523 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:14.234549 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:14.234740 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:14.234820 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:14.236668 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:14.236669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:14.236670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:14.236676 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:14.237952 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:14.238347 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:14.238348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:14.238348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:14.238404 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:14.238692 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:14.238693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:14.238693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:14.238847 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:14.238913 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:14.239163 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 11:13:14.239235 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 11:13:14.239508 3909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:14.231258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37081 (local address 127.0.98.3:36415)
0927 11:13:14.231426 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:14.231467 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:14.232027 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:14.232247 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:14.232250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:14.232755 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:14.232919 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:14.232923 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:14.233611 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:14.233612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:14.234454 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:14.234456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:14.234828 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:14.234829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:14.234948 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:14.235383 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:14.235390 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:14.236572 (+ 1182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:14.238043 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:14.238066 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:14.238068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:14.238266 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:14.238474 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:14.238475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:14.238475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:14.238605 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:14.238908 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:14.238914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:14.239040 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 11:13:14.239142 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 11:13:14.239282 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260927 11:13:14.239951 3907 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37081
I20260927 11:13:14.240018 3907 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:14.240127 3907 heartbeater.cc:511] Master 127.0.98.62:37081 requested a full tablet report, sending...
I20260927 11:13:14.240422 3448 ts_manager.cc:194] Registered new tserver with Master: f64cb34c8bcd4634bdc61d2c35502e6c (127.0.98.3:33817)
I20260927 11:13:14.240777 3448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36415
I20260927 11:13:14.242966 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 11:13:14 dist-test-slave-9hq9 krb5kdc[3403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507594, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 11:13:14 dist-test-slave-9hq9 krb5kdc[3403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507594, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:14.264490 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:14.257552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.0.98.62:37081)
0927 11:13:14.257620 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:14.257622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:14.257734 (+ 112us) server_negotiation.cc:407] Connection header received
0927 11:13:14.257825 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:14.257826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:14.257835 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:14.257907 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:14.258516 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:14.258964 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:14.259702 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:14.259908 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:14.259991 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:14.262277 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:14.262278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:14.262278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:14.262284 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:14.263597 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:14.263865 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:14.263866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:14.263866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:14.263917 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:14.264068 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:14.264069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:14.264069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:14.264183 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:14.264253 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:14.264352 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 11:13:14.264404 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 11:13:14.264727 3448 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55158
I20260927 11:13:14.265185 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 3510
I20260927 11:13:14.269914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 3644
I20260927 11:13:14.273947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 3778
I20260927 11:13:14.277803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 3416
2026-09-27T11:13:14Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1815 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:13:14 dist-test-slave-9hq9 krb5kdc[3919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:13:14 dist-test-slave-9hq9 krb5kdc[3919](info): set up 2 sockets
Sep 27 11:13:14 dist-test-slave-9hq9 krb5kdc[3919](info): commencing operation
krb5kdc: starting...
W20260927 11:13:15.297731 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:13:15 dist-test-slave-9hq9 krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:13:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:13:15.404511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38173
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:15.481353 3932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:15.481580 3932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:15.481614 3932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:15.482760 3932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:15.482825 3932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:15.482851 3932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:15.482872 3932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:15.482903 3932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:15.484302 3932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:15.484575 3932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:15.484783 3932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:15.489562 3937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:15.489634 3932 server_base.cc:1034] running on GCE node
W20260927 11:13:15.489562 3938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:15.489717 3940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:15.490118 3932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:15.490420 3932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:15.491583 3932 hybrid_clock.cc:648] HybridClock initialized: now 1790507595491573 us; error 39 us; skew 500 ppm
Sep 27 11:13:15 dist-test-slave-9hq9 krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:15.500274 3932 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:13:15.500720 3932 webserver.cc:460] Webserver started at http://127.0.98.62:39307/ using document root <none> and password file <none>
I20260927 11:13:15.500931 3932 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:15.500996 3932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:15.501140 3932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:15.502179 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "183781dfffa24a969d1cad3b407b668e"
format_stamp: "Formatted at 2026-09-27 11:13:15 on dist-test-slave-9hq9"
server_key: "11b4ca55bb53ac6ed147073877b6436d"
server_key_iv: "6bbc3a6a536daa08df2c6c6ac168bc35"
server_key_version: "encryptionkey@0"
I20260927 11:13:15.502548 3932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "183781dfffa24a969d1cad3b407b668e"
format_stamp: "Formatted at 2026-09-27 11:13:15 on dist-test-slave-9hq9"
server_key: "11b4ca55bb53ac6ed147073877b6436d"
server_key_iv: "6bbc3a6a536daa08df2c6c6ac168bc35"
server_key_version: "encryptionkey@0"
I20260927 11:13:15.503767 3932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:15.504429 3947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:15.504593 3932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:15.504686 3932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "183781dfffa24a969d1cad3b407b668e"
format_stamp: "Formatted at 2026-09-27 11:13:15 on dist-test-slave-9hq9"
server_key: "11b4ca55bb53ac6ed147073877b6436d"
server_key_iv: "6bbc3a6a536daa08df2c6c6ac168bc35"
server_key_version: "encryptionkey@0"
I20260927 11:13:15.504758 3932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:15.518561 3932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:15.520192 3932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:15.520351 3932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:15.524166 3932 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38173
I20260927 11:13:15.524194 3999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38173 every 8 connection(s)
I20260927 11:13:15.524503 3932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:15.525290 4000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:15.527403 4000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e: Bootstrap starting.
I20260927 11:13:15.527993 4000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:15.528270 4000 log.cc:824] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:15.528750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 3932
I20260927 11:13:15.528854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:15.528982 4000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e: No bootstrap required, opened a new log
I20260927 11:13:15.529030 392 external_mini_cluster.cc:1428] Setting key 3b9ee07f91798644fb6d2d125d9c6947
I20260927 11:13:15.530828 4000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183781dfffa24a969d1cad3b407b668e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38173 } }
I20260927 11:13:15.530984 4000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:15.531023 4000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 183781dfffa24a969d1cad3b407b668e, State: Initialized, Role: FOLLOWER
I20260927 11:13:15.531153 4000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183781dfffa24a969d1cad3b407b668e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38173 } }
I20260927 11:13:15.531235 4000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:15.531276 4000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:15.531325 4000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:15.532233 4000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183781dfffa24a969d1cad3b407b668e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38173 } }
I20260927 11:13:15.532346 4000 leader_election.cc:302] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 183781dfffa24a969d1cad3b407b668e; no voters:
I20260927 11:13:15.532519 4000 leader_election.cc:288] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:15.532546 4005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:15.532639 4005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [term 1 LEADER]: Becoming Leader. State: Replica: 183781dfffa24a969d1cad3b407b668e, State: Running, Role: LEADER
I20260927 11:13:15.532712 4005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183781dfffa24a969d1cad3b407b668e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38173 } }
I20260927 11:13:15.532768 4000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:15.533098 4006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "183781dfffa24a969d1cad3b407b668e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183781dfffa24a969d1cad3b407b668e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38173 } } }
I20260927 11:13:15.533162 4006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:15.533318 4007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 183781dfffa24a969d1cad3b407b668e. Latest consensus state: current_term: 1 leader_uuid: "183781dfffa24a969d1cad3b407b668e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "183781dfffa24a969d1cad3b407b668e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38173 } } }
I20260927 11:13:15.533370 4007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:15.533828 4009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:15.534325 4009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:15.536162 4009 catalog_manager.cc:1347] Generated new cluster ID: 511cf02acf354b94bec139b8718c3b28
I20260927 11:13:15.536219 4009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:13:15 dist-test-slave-9hq9 krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:15.540594 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:15.529477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59738 (local address 127.0.98.62:38173)
0927 11:13:15.529650 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:15.529658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:15.529680 (+ 22us) server_negotiation.cc:407] Connection header received
0927 11:13:15.530814 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:15.530820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:15.530862 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:15.530969 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:15.531745 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:15.532720 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:15.533797 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:15.534414 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:15.534806 (+ 392us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:15.537143 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:15.537147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:15.537147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:15.537159 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:15.539036 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:15.539391 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:15.539394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:15.539395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:15.539508 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:15.539701 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:15.539702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:15.539703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:15.539987 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:15.540141 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:15.540230 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 11:13:15.540377 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260927 11:13:15.565419 4009 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:15.565886 4009 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:15.568042 4009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 183781dfffa24a969d1cad3b407b668e: Generated new TSK 0
I20260927 11:13:15.568199 4009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:13:15.593776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38173
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:15.689409 4026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:15.689592 4026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:15.689615 4026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:15.690743 4026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:15.690788 4026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:15.690840 4026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:15.692251 4026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:15.692487 4026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:15.692693 4026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:15.697407 4034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:15.697445 4026 server_base.cc:1034] running on GCE node
W20260927 11:13:15.697435 4031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:15.697463 4032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:15.697842 4026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:15.698127 4026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:15.699265 4026 hybrid_clock.cc:648] HybridClock initialized: now 1790507595699247 us; error 34 us; skew 500 ppm
Sep 27 11:13:15 dist-test-slave-9hq9 krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:15.701571 4026 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:13:15.702020 4026 webserver.cc:460] Webserver started at http://127.0.98.1:44083/ using document root <none> and password file <none>
I20260927 11:13:15.702265 4026 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:15.702337 4026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:15.702481 4026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:15.703419 4026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "a48810cdf94541a8ae8d94925dd1db76"
format_stamp: "Formatted at 2026-09-27 11:13:15 on dist-test-slave-9hq9"
server_key: "92f9bc168cb34d7b0e501658cc1abc21"
server_key_iv: "85c89ae0108371e365822dc3f552f109"
server_key_version: "encryptionkey@0"
I20260927 11:13:15.703763 4026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a48810cdf94541a8ae8d94925dd1db76"
format_stamp: "Formatted at 2026-09-27 11:13:15 on dist-test-slave-9hq9"
server_key: "92f9bc168cb34d7b0e501658cc1abc21"
server_key_iv: "85c89ae0108371e365822dc3f552f109"
server_key_version: "encryptionkey@0"
I20260927 11:13:15.706446 4026 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 11:13:15.707445 4041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:15.707640 4026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:15.707747 4026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "a48810cdf94541a8ae8d94925dd1db76"
format_stamp: "Formatted at 2026-09-27 11:13:15 on dist-test-slave-9hq9"
server_key: "92f9bc168cb34d7b0e501658cc1abc21"
server_key_iv: "85c89ae0108371e365822dc3f552f109"
server_key_version: "encryptionkey@0"
I20260927 11:13:15.707827 4026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:15.775731 4026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:15.778142 4026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:15.778301 4026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:15.778522 4026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:15.778883 4026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:15.778926 4026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:15.778954 4026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:15.778970 4026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:15.784752 4026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43193
I20260927 11:13:15.784792 4154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43193 every 8 connection(s)
I20260927 11:13:15.785086 4026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:13:15.788970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 4026
I20260927 11:13:15.789067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:13:15.789238 392 external_mini_cluster.cc:1428] Setting key b8d3963ca6996751247a3c72e630960b
Sep 27 11:13:15 dist-test-slave-9hq9 krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:15.794968 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:15.785705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38173 (local address 127.0.98.1:50503)
0927 11:13:15.785843 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:15.785889 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:15.786913 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:15.787180 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:15.787184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:15.787433 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:15.787610 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:15.787614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:15.788316 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:15.788317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:15.789368 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:15.789371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:15.789773 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:15.789774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:15.789912 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:15.790373 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:15.790383 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:15.791909 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:15.793677 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:15.793683 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:15.793684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:15.793889 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:15.794162 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:15.794162 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:15.794162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:15.794253 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:15.794537 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:15.794540 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:15.794644 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 11:13:15.794718 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 11:13:15.794824 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260927 11:13:15.794998 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:15.785736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50503 (local address 127.0.98.62:38173)
0927 11:13:15.785797 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:15.785800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:15.785966 (+ 166us) server_negotiation.cc:407] Connection header received
0927 11:13:15.787026 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:15.787027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:15.787039 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:15.787108 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:15.787717 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:15.788214 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:15.789455 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:15.789677 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:15.789771 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:15.791993 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:15.791995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:15.791995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:15.792002 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:15.793602 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:15.793963 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:15.793964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:15.793965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:15.794029 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:15.794321 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:15.794322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:15.794323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:15.794469 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:15.794544 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:15.794776 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 11:13:15.794867 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 11:13:15.795459 4155 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38173
I20260927 11:13:15.795557 4155 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:15.795738 4155 heartbeater.cc:511] Master 127.0.98.62:38173 requested a full tablet report, sending...
I20260927 11:13:15.796277 3964 ts_manager.cc:194] Registered new tserver with Master: a48810cdf94541a8ae8d94925dd1db76 (127.0.98.1:43193)
I20260927 11:13:15.796839 3964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50503
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:13:15.813210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38173
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:15.890767 4160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:15.891041 4160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:15.891078 4160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:15.892215 4160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:15.892274 4160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:15.892318 4160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:15.893740 4160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:15.893985 4160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:15.894227 4160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:15.898805 4165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:15.898890 4166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:15.898814 4168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:15.899073 4160 server_base.cc:1034] running on GCE node
I20260927 11:13:15.899271 4160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:15.899495 4160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:15.900646 4160 hybrid_clock.cc:648] HybridClock initialized: now 1790507595900623 us; error 40 us; skew 500 ppm
Sep 27 11:13:15 dist-test-slave-9hq9 krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:15.903083 4160 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:13:15.903437 4160 webserver.cc:460] Webserver started at http://127.0.98.2:36585/ using document root <none> and password file <none>
I20260927 11:13:15.903606 4160 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:15.903644 4160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:15.903725 4160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:15.904644 4160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "2124ac7d21494f44b1e3306fb62c6e89"
format_stamp: "Formatted at 2026-09-27 11:13:15 on dist-test-slave-9hq9"
server_key: "69b5e00058026474fe11fd87abf004f8"
server_key_iv: "4c2e49dca31244c9c14133422da1e8f8"
server_key_version: "encryptionkey@0"
I20260927 11:13:15.904934 4160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2124ac7d21494f44b1e3306fb62c6e89"
format_stamp: "Formatted at 2026-09-27 11:13:15 on dist-test-slave-9hq9"
server_key: "69b5e00058026474fe11fd87abf004f8"
server_key_iv: "4c2e49dca31244c9c14133422da1e8f8"
server_key_version: "encryptionkey@0"
I20260927 11:13:15.906177 4160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:13:15.906858 4175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:15.907006 4160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:15.907097 4160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "2124ac7d21494f44b1e3306fb62c6e89"
format_stamp: "Formatted at 2026-09-27 11:13:15 on dist-test-slave-9hq9"
server_key: "69b5e00058026474fe11fd87abf004f8"
server_key_iv: "4c2e49dca31244c9c14133422da1e8f8"
server_key_version: "encryptionkey@0"
I20260927 11:13:15.907162 4160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:15.919379 4160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:15.921535 4160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:15.921638 4160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:15.921799 4160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:15.922036 4160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:15.922130 4160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:15.922154 4160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:15.922168 4160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:15.927636 4160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41763
I20260927 11:13:15.927706 4288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41763 every 8 connection(s)
I20260927 11:13:15.927989 4160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:13:15 dist-test-slave-9hq9 krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:15.936632 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:15.928680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46283 (local address 127.0.98.62:38173)
0927 11:13:15.928745 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:15.928747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:15.928959 (+ 212us) server_negotiation.cc:407] Connection header received
0927 11:13:15.929833 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:15.929834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:15.929843 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:15.929904 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:15.930504 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:15.930986 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:15.931884 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:15.932061 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:15.932109 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:15.934221 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:15.934222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:15.934222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:15.934228 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:15.935579 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:15.935923 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:15.935924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:15.935924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:15.935969 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:15.936176 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:15.936177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:15.936177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:15.936268 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:15.936323 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:15.936519 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 11:13:15.936573 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 11:13:15.936882 4291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:15.928669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38173 (local address 127.0.98.2:46283)
0927 11:13:15.928841 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:15.928879 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:15.929750 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:15.929957 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:15.929960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:15.930250 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:15.930434 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:15.930439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:15.931058 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:15.931059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:15.931816 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:15.931817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:15.932164 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:15.932165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:15.932285 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:15.932649 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:15.932657 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:15.934140 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:15.935654 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:15.935659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:15.935661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:15.935860 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:15.936026 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:15.936027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:15.936027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:15.936124 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:15.936321 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:15.936324 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:15.936425 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 11:13:15.936508 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 11:13:15.936617 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":109,"threads_started":1}
I20260927 11:13:15.937354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 4160
I20260927 11:13:15.937430 4289 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38173
I20260927 11:13:15.937463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:13:15.937521 4289 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:15.937654 392 external_mini_cluster.cc:1428] Setting key 439fca2a72284e5ed43bd7ad81da2ed2
I20260927 11:13:15.937703 4289 heartbeater.cc:511] Master 127.0.98.62:38173 requested a full tablet report, sending...
I20260927 11:13:15.938109 3963 ts_manager.cc:194] Registered new tserver with Master: 2124ac7d21494f44b1e3306fb62c6e89 (127.0.98.2:41763)
I20260927 11:13:15.938535 3963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46283
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:13:15.960776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38173
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:16.038615 4294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:16.038808 4294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:16.038842 4294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:16.039942 4294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:16.040002 4294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:16.040045 4294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:16.041428 4294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:16.041661 4294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:16.041868 4294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:13:16.046633 4294 server_base.cc:1034] running on GCE node
W20260927 11:13:16.046689 4300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:16.046559 4302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:16.046574 4299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:16.046944 4294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:16.047180 4294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:16.048337 4294 hybrid_clock.cc:648] HybridClock initialized: now 1790507596048326 us; error 28 us; skew 500 ppm
Sep 27 11:13:16 dist-test-slave-9hq9 krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:16.050601 4294 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:13:16.051024 4294 webserver.cc:460] Webserver started at http://127.0.98.3:35335/ using document root <none> and password file <none>
I20260927 11:13:16.051219 4294 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:16.051285 4294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:16.051429 4294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:16.052388 4294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "db0464cb3fec43899c201935d1795afe"
format_stamp: "Formatted at 2026-09-27 11:13:16 on dist-test-slave-9hq9"
server_key: "d9bb275bd9e28460b6d5ecf30773b1cd"
server_key_iv: "96d2b8a4e4f838abef079b7f0f8dd313"
server_key_version: "encryptionkey@0"
I20260927 11:13:16.052714 4294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "db0464cb3fec43899c201935d1795afe"
format_stamp: "Formatted at 2026-09-27 11:13:16 on dist-test-slave-9hq9"
server_key: "d9bb275bd9e28460b6d5ecf30773b1cd"
server_key_iv: "96d2b8a4e4f838abef079b7f0f8dd313"
server_key_version: "encryptionkey@0"
I20260927 11:13:16.053918 4294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:13:16.054642 4309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:16.054801 4294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:16.054889 4294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "db0464cb3fec43899c201935d1795afe"
format_stamp: "Formatted at 2026-09-27 11:13:16 on dist-test-slave-9hq9"
server_key: "d9bb275bd9e28460b6d5ecf30773b1cd"
server_key_iv: "96d2b8a4e4f838abef079b7f0f8dd313"
server_key_version: "encryptionkey@0"
I20260927 11:13:16.054970 4294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:16.063503 4294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:16.065089 4294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:16.065234 4294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:16.065430 4294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:16.065687 4294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:16.065753 4294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:16.065793 4294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:16.065821 4294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:16.071715 4294 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38689
I20260927 11:13:16.071748 4422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38689 every 8 connection(s)
I20260927 11:13:16.072016 4294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:13:16.074540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 4294
I20260927 11:13:16.074633 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:13:16.074795 392 external_mini_cluster.cc:1428] Setting key f3910d71f3c8ae4a9cffc6d92d599be7
Sep 27 11:13:16 dist-test-slave-9hq9 krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:16.080514 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:16.072655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47129 (local address 127.0.98.62:38173)
0927 11:13:16.072713 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:16.072715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:16.072903 (+ 188us) server_negotiation.cc:407] Connection header received
0927 11:13:16.073702 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:16.073703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:16.073712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:16.073774 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:16.074342 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:16.075097 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:16.076252 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:16.076425 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:16.076476 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:16.078406 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:16.078407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:16.078407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:16.078412 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:16.079525 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:16.079843 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:16.079844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:16.079844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:16.079887 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:16.080071 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:16.080072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:16.080072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:16.080155 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:16.080201 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:16.080401 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 11:13:16.080455 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 11:13:16.080632 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:16.072635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38173 (local address 127.0.98.3:47129)
0927 11:13:16.072789 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:16.072825 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:16.073624 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:16.073823 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:16.073826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:16.074042 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:16.074267 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:16.074271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:16.075202 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:16.075203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:16.076160 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:16.076162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:16.076502 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:16.076502 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:16.076590 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:16.077063 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:16.077073 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:16.078335 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:16.079589 (+ 1254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:16.079594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:16.079595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:16.079782 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:16.079939 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:16.079940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:16.079940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:16.080022 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:16.080204 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:16.080208 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:16.080304 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 11:13:16.080375 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:13:16.080470 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":94,"threads_started":1}
I20260927 11:13:16.081074 4423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38173
I20260927 11:13:16.081166 4423 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:16.081319 4423 heartbeater.cc:511] Master 127.0.98.62:38173 requested a full tablet report, sending...
I20260927 11:13:16.081648 3963 ts_manager.cc:194] Registered new tserver with Master: db0464cb3fec43899c201935d1795afe (127.0.98.3:38689)
I20260927 11:13:16.081990 3963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47129
I20260927 11:13:16.086390 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 11:13:16 dist-test-slave-9hq9 krb5kdc[3919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 11:13:16 dist-test-slave-9hq9 krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:13:16.107267 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:16.100307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46308 (local address 127.0.98.1:43193)
0927 11:13:16.100458 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:16.100463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:16.100472 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:13:16.100492 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:16.100494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:16.100506 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:16.100566 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:16.101041 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:16.101679 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:16.102436 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:16.102715 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:16.102766 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:16.104691 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:16.104697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:16.104697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:16.104702 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:16.106275 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:16.106566 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:16.106568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:16.106568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:16.106628 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:16.106782 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:16.106782 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:16.106783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:16.106895 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:16.106986 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:16.107096 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 11:13:16.107151 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
Sep 27 11:13:16 dist-test-slave-9hq9 krb5kdc[3919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:16.115450 4004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:16.108446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59754 (local address 127.0.98.62:38173)
0927 11:13:16.108499 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:16.108505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:16.108513 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:13:16.108529 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:16.108530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:16.108537 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:16.108581 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:16.108993 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:16.109434 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:16.110154 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:16.110322 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:16.110370 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:16.113139 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:16.113140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:16.113141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:16.113147 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:16.114575 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:16.114871 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:16.114872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:16.114872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:16.114917 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:16.115056 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:16.115056 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:16.115056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:16.115165 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:16.115230 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:16.115313 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 11:13:16.115363 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 11:13:16.115657 3963 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59754
I20260927 11:13:16.115967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 4026
I20260927 11:13:16.120879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 4160
I20260927 11:13:16.125039 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 4294
I20260927 11:13:16.129042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 3932
2026-09-27T11:13:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1851 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:13:16 dist-test-slave-9hq9 krb5kdc[4435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:13:16 dist-test-slave-9hq9 krb5kdc[4435](info): set up 2 sockets
Sep 27 11:13:16 dist-test-slave-9hq9 krb5kdc[4435](info): commencing operation
krb5kdc: starting...
W20260927 11:13:17.150020 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:13:17 dist-test-slave-9hq9 krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:13:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:13:17.258919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35841
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:17.334354 4448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:17.334563 4448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:17.334596 4448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:17.335724 4448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:17.335803 4448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:17.335826 4448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:17.335855 4448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:17.335883 4448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:17.337338 4448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:17.337597 4448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:17.337805 4448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:17.342342 4453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:17.342345 4454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:17.342417 4448 server_base.cc:1034] running on GCE node
W20260927 11:13:17.342338 4456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:17.342760 4448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:17.343071 4448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:17.344215 4448 hybrid_clock.cc:648] HybridClock initialized: now 1790507597344206 us; error 27 us; skew 500 ppm
Sep 27 11:13:17 dist-test-slave-9hq9 krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:17.346598 4448 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:13:17.346946 4448 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:17.347012 4448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:17.347163 4448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:17.348140 4448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "36a88adc6627438e8e432461826bc3ec"
format_stamp: "Formatted at 2026-09-27 11:13:17 on dist-test-slave-9hq9"
server_key: "1cd382256a427068d47194c2dd032849"
server_key_iv: "2429671ad439f012ef4ed69918a3808b"
server_key_version: "encryptionkey@0"
I20260927 11:13:17.348497 4448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "36a88adc6627438e8e432461826bc3ec"
format_stamp: "Formatted at 2026-09-27 11:13:17 on dist-test-slave-9hq9"
server_key: "1cd382256a427068d47194c2dd032849"
server_key_iv: "2429671ad439f012ef4ed69918a3808b"
server_key_version: "encryptionkey@0"
I20260927 11:13:17.349756 4448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:13:17.350483 4462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:17.350644 4448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:17.350735 4448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "36a88adc6627438e8e432461826bc3ec"
format_stamp: "Formatted at 2026-09-27 11:13:17 on dist-test-slave-9hq9"
server_key: "1cd382256a427068d47194c2dd032849"
server_key_iv: "2429671ad439f012ef4ed69918a3808b"
server_key_version: "encryptionkey@0"
I20260927 11:13:17.350811 4448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:17.376111 4448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:17.377641 4448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:17.377785 4448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:17.381373 4448 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35841
I20260927 11:13:17.381398 4514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35841 every 8 connection(s)
I20260927 11:13:17.381654 4448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:17.382517 4515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:17.383129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 4448
I20260927 11:13:17.383239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:17.383407 392 external_mini_cluster.cc:1428] Setting key 36f9a80f40685a42fe5bbee8f7290263
I20260927 11:13:17.385360 4515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec: Bootstrap starting.
I20260927 11:13:17.386089 4515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:17.386368 4515 log.cc:824] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:17.387043 4515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec: No bootstrap required, opened a new log
I20260927 11:13:17.389045 4515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a88adc6627438e8e432461826bc3ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35841 } }
I20260927 11:13:17.389182 4515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:17.389225 4515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a88adc6627438e8e432461826bc3ec, State: Initialized, Role: FOLLOWER
I20260927 11:13:17.389361 4515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a88adc6627438e8e432461826bc3ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35841 } }
I20260927 11:13:17.389452 4515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:17.389493 4515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:17.389541 4515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:17.390192 4515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a88adc6627438e8e432461826bc3ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35841 } }
Sep 27 11:13:17 dist-test-slave-9hq9 krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:17.390296 4515 leader_election.cc:302] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36a88adc6627438e8e432461826bc3ec; no voters:
I20260927 11:13:17.390447 4515 leader_election.cc:288] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:17.390496 4520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:17.390683 4520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [term 1 LEADER]: Becoming Leader. State: Replica: 36a88adc6627438e8e432461826bc3ec, State: Running, Role: LEADER
I20260927 11:13:17.390687 4515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:17.390797 4520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a88adc6627438e8e432461826bc3ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35841 } }
I20260927 11:13:17.391197 4520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36a88adc6627438e8e432461826bc3ec. Latest consensus state: current_term: 1 leader_uuid: "36a88adc6627438e8e432461826bc3ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a88adc6627438e8e432461826bc3ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35841 } } }
I20260927 11:13:17.391208 4521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36a88adc6627438e8e432461826bc3ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a88adc6627438e8e432461826bc3ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35841 } } }
I20260927 11:13:17.391695 4521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:17.391346 4520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:17.393249 4532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:17.393924 4532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:17.394243 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:17.383874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.0.98.62:35841)
0927 11:13:17.384039 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:17.384046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:17.384078 (+ 32us) server_negotiation.cc:407] Connection header received
0927 11:13:17.384629 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:17.384638 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:17.384698 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:17.384837 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:17.385552 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.386933 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.387689 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.388279 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.388380 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:17.390865 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:17.390869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:17.390870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:17.390881 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:17.392616 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:17.392982 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:17.392985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:17.392985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:17.393109 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:17.393393 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:17.393394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:17.393395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:17.393643 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:17.393790 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:17.393899 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 11:13:17.394027 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260927 11:13:17.395416 4532 catalog_manager.cc:1347] Generated new cluster ID: 138e36f63df94441a246fb037204b874
I20260927 11:13:17.395466 4532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:13:17.406212 4532 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:17.406703 4532 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:17.411942 4532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec: Generated new TSK 0
I20260927 11:13:17.412104 4532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:13:17.444175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35841
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:17.521281 4541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:17.521447 4541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:17.521468 4541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:17.522580 4541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:17.522619 4541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:17.522675 4541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:17.524147 4541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:17.524312 4541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:17.524523 4541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:17.529094 4549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:17.529210 4541 server_base.cc:1034] running on GCE node
W20260927 11:13:17.529184 4546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:17.529278 4547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:17.529567 4541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:17.529810 4541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:17.530952 4541 hybrid_clock.cc:648] HybridClock initialized: now 1790507597530945 us; error 31 us; skew 500 ppm
Sep 27 11:13:17 dist-test-slave-9hq9 krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:17.533183 4541 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:13:17.533447 4541 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:17.533488 4541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:17.533567 4541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:17.534514 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "f3f809fa41cb4339b213eb49e393a915"
format_stamp: "Formatted at 2026-09-27 11:13:17 on dist-test-slave-9hq9"
server_key: "877c070a13daf257439979037f1d7b9b"
server_key_iv: "dc636277207fdcd431b369890c09eb58"
server_key_version: "encryptionkey@0"
I20260927 11:13:17.534796 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f3f809fa41cb4339b213eb49e393a915"
format_stamp: "Formatted at 2026-09-27 11:13:17 on dist-test-slave-9hq9"
server_key: "877c070a13daf257439979037f1d7b9b"
server_key_iv: "dc636277207fdcd431b369890c09eb58"
server_key_version: "encryptionkey@0"
I20260927 11:13:17.535846 4541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:13:17.536401 4555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:17.536552 4541 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:17.536638 4541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "f3f809fa41cb4339b213eb49e393a915"
format_stamp: "Formatted at 2026-09-27 11:13:17 on dist-test-slave-9hq9"
server_key: "877c070a13daf257439979037f1d7b9b"
server_key_iv: "dc636277207fdcd431b369890c09eb58"
server_key_version: "encryptionkey@0"
I20260927 11:13:17.536715 4541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:17.551281 4541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:17.552949 4541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:17.553117 4541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:17.553320 4541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:17.553601 4541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:17.553653 4541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:17.553699 4541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:17.553727 4541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:17.559051 4541 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38219
I20260927 11:13:17.559113 4668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38219 every 8 connection(s)
I20260927 11:13:17.559365 4541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:13:17 dist-test-slave-9hq9 krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:17.567806 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:17.560029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34385 (local address 127.0.98.62:35841)
0927 11:13:17.560083 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:17.560085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:17.560274 (+ 189us) server_negotiation.cc:407] Connection header received
0927 11:13:17.560743 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:17.560744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:17.560755 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:17.560816 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:17.561645 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.562236 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.563208 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.563418 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.563499 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:17.565340 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:17.565341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:17.565341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:17.565349 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:17.566523 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:17.566906 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:17.566907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:17.566908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:17.566967 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:17.567209 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:17.567210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:17.567210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:17.567316 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:17.567387 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:17.567639 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 11:13:17.567719 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:13:17.567975 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:17.559996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35841 (local address 127.0.98.1:34385)
0927 11:13:17.560161 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:17.560201 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:17.560642 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:17.560876 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:17.560879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:17.561109 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:17.561551 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:17.561555 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.562334 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.562335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:17.563104 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:17.563106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.563508 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.563509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:17.563625 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:17.563998 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:17.564005 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:17.565247 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:17.566617 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:17.566624 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:17.566626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:17.566831 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:17.567034 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:17.567034 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:17.567035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:17.567133 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:17.567388 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:17.567394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:17.567516 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 11:13:17.567618 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 11:13:17.567765 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260927 11:13:17.568480 4669 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35841
I20260927 11:13:17.568537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 4541
I20260927 11:13:17.568589 4669 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:17.568635 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:13:17.568742 4669 heartbeater.cc:511] Master 127.0.98.62:35841 requested a full tablet report, sending...
I20260927 11:13:17.568828 392 external_mini_cluster.cc:1428] Setting key ad562d2039f0d87d69b35329553751b1
I20260927 11:13:17.569231 4479 ts_manager.cc:194] Registered new tserver with Master: f3f809fa41cb4339b213eb49e393a915 (127.0.98.1:38219)
I20260927 11:13:17.569841 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34385
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:13:17.590608 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35841
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:17.666733 4674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:17.666903 4674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:17.666925 4674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:17.667968 4674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:17.668000 4674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:17.668026 4674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:17.669369 4674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:17.669520 4674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:17.669670 4674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:17.674229 4682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:17.674250 4679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:17.674250 4680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:17.674377 4674 server_base.cc:1034] running on GCE node
I20260927 11:13:17.674640 4674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:17.674891 4674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:17.676041 4674 hybrid_clock.cc:648] HybridClock initialized: now 1790507597676033 us; error 26 us; skew 500 ppm
Sep 27 11:13:17 dist-test-slave-9hq9 krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:17.678275 4674 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:13:17.678577 4674 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:17.678646 4674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:17.678799 4674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:17.679723 4674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea"
format_stamp: "Formatted at 2026-09-27 11:13:17 on dist-test-slave-9hq9"
server_key: "4a0c024c1b7089f0fc8437897844c9f6"
server_key_iv: "faa511034b43f57bf24b6d6ae73f6c0d"
server_key_version: "encryptionkey@0"
I20260927 11:13:17.680048 4674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea"
format_stamp: "Formatted at 2026-09-27 11:13:17 on dist-test-slave-9hq9"
server_key: "4a0c024c1b7089f0fc8437897844c9f6"
server_key_iv: "faa511034b43f57bf24b6d6ae73f6c0d"
server_key_version: "encryptionkey@0"
I20260927 11:13:17.681229 4674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:13:17.681874 4688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:17.682035 4674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:17.682163 4674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea"
format_stamp: "Formatted at 2026-09-27 11:13:17 on dist-test-slave-9hq9"
server_key: "4a0c024c1b7089f0fc8437897844c9f6"
server_key_iv: "faa511034b43f57bf24b6d6ae73f6c0d"
server_key_version: "encryptionkey@0"
I20260927 11:13:17.682248 4674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:17.707695 4674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:17.710984 4674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:17.711151 4674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:17.711382 4674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:17.711803 4674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:17.711874 4674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:17.711926 4674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:17.711954 4674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:17.718142 4674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41745
I20260927 11:13:17.718187 4801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41745 every 8 connection(s)
I20260927 11:13:17.718436 4674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:13:17 dist-test-slave-9hq9 krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:17.724901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 4674
I20260927 11:13:17.724999 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:13:17.725183 392 external_mini_cluster.cc:1428] Setting key 60262866315aa3dad6ae1da3526ee3dc
I20260927 11:13:17.726740 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:17.719078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35841 (local address 127.0.98.2:43311)
0927 11:13:17.719217 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:17.719248 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:17.719677 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:17.719860 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:17.719863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:17.720087 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:17.720559 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:17.720564 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.721180 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.721181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:17.721924 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:17.721925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.722302 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.722303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:17.722388 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:17.722755 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:17.722762 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:17.724028 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:17.725425 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:17.725431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:17.725433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:17.725716 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:17.725895 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:17.725895 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:17.725895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:17.726007 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:17.726300 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:17.726304 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:17.726406 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 11:13:17.726487 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 11:13:17.726591 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":84,"threads_started":1}
I20260927 11:13:17.726754 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:17.719085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43311 (local address 127.0.98.62:35841)
0927 11:13:17.719135 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:17.719137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:17.719334 (+ 197us) server_negotiation.cc:407] Connection header received
0927 11:13:17.719751 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:17.719752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:17.719761 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:17.719805 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:17.720629 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.721106 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.721990 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.722228 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.722288 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:17.724109 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:17.724110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:17.724110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:17.724117 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:17.725349 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:17.725777 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:17.725777 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:17.725778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:17.725835 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:17.726095 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:17.726096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:17.726097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:17.726232 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:17.726305 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:17.726527 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 11:13:17.726611 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:13:17.727471 4802 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35841
I20260927 11:13:17.727567 4802 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:17.727715 4802 heartbeater.cc:511] Master 127.0.98.62:35841 requested a full tablet report, sending...
I20260927 11:13:17.728031 4479 ts_manager.cc:194] Registered new tserver with Master: 9cfdaad9f7a94dbba3208ef70d7d89ea (127.0.98.2:41745)
I20260927 11:13:17.728417 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43311
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:13:17.748464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35841
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:17.826606 4807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:17.826781 4807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:17.826803 4807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:17.827919 4807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:17.827958 4807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:17.827984 4807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:17.829424 4807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:17.829586 4807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:17.829798 4807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:17.834321 4813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:17.834450 4815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:17.834358 4812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:17.834570 4807 server_base.cc:1034] running on GCE node
I20260927 11:13:17.834812 4807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:17.834981 4807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:17.836110 4807 hybrid_clock.cc:648] HybridClock initialized: now 1790507597836087 us; error 38 us; skew 500 ppm
Sep 27 11:13:17 dist-test-slave-9hq9 krb5kdc[4435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:17.838471 4807 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:13:17.838730 4807 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:17.838769 4807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:17.838850 4807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:17.839785 4807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "d326d46da41e4cac82607b8d7290ea9d"
format_stamp: "Formatted at 2026-09-27 11:13:17 on dist-test-slave-9hq9"
server_key: "f2b9694f81bcaa29af3295c1e0ae74bd"
server_key_iv: "edc47c764aef8251cd6a3a8efc831ed1"
server_key_version: "encryptionkey@0"
I20260927 11:13:17.840077 4807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d326d46da41e4cac82607b8d7290ea9d"
format_stamp: "Formatted at 2026-09-27 11:13:17 on dist-test-slave-9hq9"
server_key: "f2b9694f81bcaa29af3295c1e0ae74bd"
server_key_iv: "edc47c764aef8251cd6a3a8efc831ed1"
server_key_version: "encryptionkey@0"
I20260927 11:13:17.841243 4807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:13:17.841830 4821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:17.842008 4807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:17.842115 4807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "d326d46da41e4cac82607b8d7290ea9d"
format_stamp: "Formatted at 2026-09-27 11:13:17 on dist-test-slave-9hq9"
server_key: "f2b9694f81bcaa29af3295c1e0ae74bd"
server_key_iv: "edc47c764aef8251cd6a3a8efc831ed1"
server_key_version: "encryptionkey@0"
I20260927 11:13:17.842191 4807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:17.852028 4807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:17.853854 4807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:17.853996 4807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:17.854235 4807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:17.854465 4807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:17.854494 4807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:17.854514 4807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:17.854527 4807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:17.859895 4807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41133
I20260927 11:13:17.859948 4934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41133 every 8 connection(s)
I20260927 11:13:17.860220 4807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:13:17.862756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 4807
I20260927 11:13:17.862864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:13:17.863070 392 external_mini_cluster.cc:1428] Setting key d8934365ab9680038518bfebca845e97
Sep 27 11:13:17 dist-test-slave-9hq9 krb5kdc[4435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:17.868846 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:17.860830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54575 (local address 127.0.98.62:35841)
0927 11:13:17.860895 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:17.860896 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:17.861058 (+ 162us) server_negotiation.cc:407] Connection header received
0927 11:13:17.861481 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:17.861482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:17.861491 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:17.861541 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:17.862455 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.863230 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.864224 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.864414 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.864487 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:17.866379 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:17.866380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:17.866380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:17.866387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:17.867565 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:17.867962 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:17.867963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:17.867964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:17.868019 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:17.868248 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:17.868249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:17.868249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:17.868357 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:17.868429 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:17.868680 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 11:13:17.868752 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 11:13:17.869015 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:17.860823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35841 (local address 127.0.98.3:54575)
0927 11:13:17.860969 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:17.860999 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:17.861402 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:17.861588 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:17.861591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:17.861815 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:17.862376 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:17.862382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.863331 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.863333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:17.864104 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:17.864106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.864496 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.864497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:17.864606 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:17.865029 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:17.865036 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:17.866287 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:17.867659 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:17.867665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:17.867668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:17.867875 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:17.868083 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:17.868083 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:17.868084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:17.868182 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:17.868421 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:17.868426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:17.868549 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 11:13:17.868648 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 11:13:17.868784 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":92,"threads_started":1}
I20260927 11:13:17.869418 4935 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35841
I20260927 11:13:17.869508 4935 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:17.869652 4935 heartbeater.cc:511] Master 127.0.98.62:35841 requested a full tablet report, sending...
I20260927 11:13:17.869972 4479 ts_manager.cc:194] Registered new tserver with Master: d326d46da41e4cac82607b8d7290ea9d (127.0.98.3:41133)
I20260927 11:13:17.870359 4479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54575
I20260927 11:13:17.874574 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:13:17.881326 4518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:17.875758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.0.98.62:35841)
0927 11:13:17.875826 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:17.875827 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:17.875934 (+ 107us) server_negotiation.cc:407] Connection header received
0927 11:13:17.876008 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:17.876009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:17.876019 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:17.876072 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:17.876673 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.877112 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.877837 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.878026 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.878127 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:17.879199 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:17.879200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:17.879201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:17.879207 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:17.880210 (+ 1003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:17.880543 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:17.880544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:17.880544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:17.880616 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:17.880819 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:17.880820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:17.880821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:17.880951 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:17.881051 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:17.881167 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 11:13:17.881237 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 11:13:17.882735 4479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:13:17.884191 4479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:17.889914 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:17.886185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41133 (local address 127.0.0.1:35138)
0927 11:13:17.886383 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:17.886392 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:17.886432 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:17.886633 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:17.886635 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:17.886636 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:17.886722 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:17.886724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.887723 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.887724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:17.888865 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:17.888868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.889653 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.889654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:17.889727 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:17.889738 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 11:13:17.889803 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 11:13:17.889851 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":60,"threads_started":1}
I20260927 11:13:17.889923 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:17.886339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35138 (local address 127.0.98.3:41133)
0927 11:13:17.886481 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:17.886486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:17.886496 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:13:17.886530 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:17.886531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:17.886546 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:17.886632 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:17.886779 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.887629 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.889006 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.889565 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.889650 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:17.889701 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:17.889824 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 11:13:17.889856 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260927 11:13:17.890233 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:17.886156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38219 (local address 127.0.0.1:32926)
0927 11:13:17.886300 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:17.886333 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:17.886394 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:17.887000 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:17.887000 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:17.887002 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:17.887039 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:17.887041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.888241 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.888242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:17.889069 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:17.889071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.889977 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.889978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:17.890039 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:17.890042 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:17.890088 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 11:13:17.890151 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":52,"threads_started":1}
I20260927 11:13:17.890444 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:17.886219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.0.98.2:41745)
0927 11:13:17.886656 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:17.886660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:17.886668 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:13:17.886705 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:17.886708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:17.886722 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:17.886792 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:17.886997 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.888093 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.889300 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.889969 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.890026 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:17.890302 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:17.890330 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 11:13:17.890374 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":72,"threads_started":1}
I20260927 11:13:17.890511 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:17.886241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.0.98.1:38219)
0927 11:13:17.886759 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:17.886764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:17.886773 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:13:17.886829 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:17.886831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:17.886843 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:17.886914 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:17.887122 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.888162 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.889129 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.889826 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.889893 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:17.889941 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:17.890386 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 11:13:17.890426 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":69,"threads_started":1}
I20260927 11:13:17.890691 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:17.886354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41745 (local address 127.0.0.1:39822)
0927 11:13:17.886471 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:17.886481 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:17.886533 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:17.886877 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:17.886878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:17.886881 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:17.886921 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:17.886922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.888199 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.888200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:17.889216 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:17.889218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:17.890111 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:17.890112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:17.890159 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:17.890162 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:17.890202 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 11:13:17.890229 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260927 11:13:17.890894 4869 tablet_service.cc:1467] Processing CreateTablet for tablet f90ce159e0ec40d999841c8498b464a4 (DEFAULT_TABLE table=test-table [id=2519bd77dbd740a19121722126f86b2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:17.891273 4869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f90ce159e0ec40d999841c8498b464a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:17.891402 4736 tablet_service.cc:1467] Processing CreateTablet for tablet f90ce159e0ec40d999841c8498b464a4 (DEFAULT_TABLE table=test-table [id=2519bd77dbd740a19121722126f86b2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:17.891662 4736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f90ce159e0ec40d999841c8498b464a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:17.893724 4949 tablet_bootstrap.cc:492] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d: Bootstrap starting.
I20260927 11:13:17.894431 4949 tablet_bootstrap.cc:654] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:17.894357 4603 tablet_service.cc:1467] Processing CreateTablet for tablet f90ce159e0ec40d999841c8498b464a4 (DEFAULT_TABLE table=test-table [id=2519bd77dbd740a19121722126f86b2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:17.894657 4603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f90ce159e0ec40d999841c8498b464a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:17.895008 4949 log.cc:824] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:17.895720 4949 tablet_bootstrap.cc:492] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d: No bootstrap required, opened a new log
I20260927 11:13:17.895785 4949 ts_tablet_manager.cc:1397] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:13:17.897759 4952 tablet_bootstrap.cc:492] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915: Bootstrap starting.
I20260927 11:13:17.897691 4949 raft_consensus.cc:348] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41745 } } peers { permanent_uuid: "f3f809fa41cb4339b213eb49e393a915" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38219 } } peers { permanent_uuid: "d326d46da41e4cac82607b8d7290ea9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41133 } }
I20260927 11:13:17.897862 4949 raft_consensus.cc:374] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:17.897907 4949 raft_consensus.cc:729] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d326d46da41e4cac82607b8d7290ea9d, State: Initialized, Role: FOLLOWER
I20260927 11:13:17.897934 4950 tablet_bootstrap.cc:492] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea: Bootstrap starting.
I20260927 11:13:17.898033 4949 consensus_queue.cc:260] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41745 } } peers { permanent_uuid: "f3f809fa41cb4339b213eb49e393a915" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38219 } } peers { permanent_uuid: "d326d46da41e4cac82607b8d7290ea9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41133 } }
I20260927 11:13:17.898322 4935 heartbeater.cc:503] Master 127.0.98.62:35841 was elected leader, sending a full tablet report...
I20260927 11:13:17.898329 4949 ts_tablet_manager.cc:1428] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:13:17.898435 4952 tablet_bootstrap.cc:654] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:17.898562 4950 tablet_bootstrap.cc:654] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:17.898960 4952 log.cc:824] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:17.899092 4950 log.cc:824] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:17.899648 4952 tablet_bootstrap.cc:492] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915: No bootstrap required, opened a new log
I20260927 11:13:17.899725 4952 ts_tablet_manager.cc:1397] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 11:13:17.899819 4950 tablet_bootstrap.cc:492] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea: No bootstrap required, opened a new log
I20260927 11:13:17.899880 4950 ts_tablet_manager.cc:1397] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:17.901002 4952 raft_consensus.cc:348] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41745 } } peers { permanent_uuid: "f3f809fa41cb4339b213eb49e393a915" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38219 } } peers { permanent_uuid: "d326d46da41e4cac82607b8d7290ea9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41133 } }
I20260927 11:13:17.901153 4952 raft_consensus.cc:374] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:17.901197 4952 raft_consensus.cc:729] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f809fa41cb4339b213eb49e393a915, State: Initialized, Role: FOLLOWER
I20260927 11:13:17.901121 4950 raft_consensus.cc:348] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41745 } } peers { permanent_uuid: "f3f809fa41cb4339b213eb49e393a915" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38219 } } peers { permanent_uuid: "d326d46da41e4cac82607b8d7290ea9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41133 } }
I20260927 11:13:17.901289 4950 raft_consensus.cc:374] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:17.901302 4952 consensus_queue.cc:260] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41745 } } peers { permanent_uuid: "f3f809fa41cb4339b213eb49e393a915" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38219 } } peers { permanent_uuid: "d326d46da41e4cac82607b8d7290ea9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41133 } }
I20260927 11:13:17.901332 4950 raft_consensus.cc:729] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cfdaad9f7a94dbba3208ef70d7d89ea, State: Initialized, Role: FOLLOWER
I20260927 11:13:17.901438 4950 consensus_queue.cc:260] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41745 } } peers { permanent_uuid: "f3f809fa41cb4339b213eb49e393a915" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38219 } } peers { permanent_uuid: "d326d46da41e4cac82607b8d7290ea9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41133 } }
I20260927 11:13:17.901544 4669 heartbeater.cc:503] Master 127.0.98.62:35841 was elected leader, sending a full tablet report...
I20260927 11:13:17.901548 4952 ts_tablet_manager.cc:1428] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 11:13:17.901733 4950 ts_tablet_manager.cc:1428] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:13:17.901770 4802 heartbeater.cc:503] Master 127.0.98.62:35841 was elected leader, sending a full tablet report...
W20260927 11:13:17.969031 4803 tablet.cc:2369] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:13:18.034300 4956 raft_consensus.cc:482] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:18.034410 4956 raft_consensus.cc:504] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41745 } } peers { permanent_uuid: "f3f809fa41cb4339b213eb49e393a915" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38219 } } peers { permanent_uuid: "d326d46da41e4cac82607b8d7290ea9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41133 } }
I20260927 11:13:18.034822 4956 leader_election.cc:288] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cfdaad9f7a94dbba3208ef70d7d89ea (127.0.98.2:41745), d326d46da41e4cac82607b8d7290ea9d (127.0.98.3:41133)
I20260927 11:13:18.038682 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.035060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53099 (local address 127.0.98.3:41133)
0927 11:13:18.035133 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.035135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.035299 (+ 164us) server_negotiation.cc:407] Connection header received
0927 11:13:18.035332 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.035333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.035343 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.035385 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:18.035532 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.036203 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.037776 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.038340 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.038412 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:18.038421 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:18.038575 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 11:13:18.038613 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:13:18.038682 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.035047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51271 (local address 127.0.98.2:41745)
0927 11:13:18.035093 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.035095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.035107 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:13:18.035173 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.035174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.035185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.035245 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:18.035444 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.036199 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.037777 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.038340 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.038412 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:18.038421 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:18.038583 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 11:13:18.038616 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:13:18.038725 4671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.034989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41745 (local address 127.0.98.1:51271)
0927 11:13:18.035048 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:18.035060 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:18.035101 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:18.035272 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:18.035273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:18.035275 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:18.035346 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:18.035348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.036293 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.036294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:18.037690 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:18.037693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.038420 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.038421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:18.038503 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:18.038505 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:13:18.038557 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 11:13:18.038604 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":89}
I20260927 11:13:18.038733 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.035003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41133 (local address 127.0.98.1:53099)
0927 11:13:18.035201 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:18.035209 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:18.035285 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:18.035409 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:18.035410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:18.035411 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:18.035467 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:18.035468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.036293 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.036294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:18.037690 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:18.037693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.038420 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.038421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:18.038495 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:18.038499 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:18.038557 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 11:13:18.038604 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":104,"thread_start_us":127,"threads_started":1}
I20260927 11:13:18.039152 4756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90ce159e0ec40d999841c8498b464a4" candidate_uuid: "f3f809fa41cb4339b213eb49e393a915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea" is_pre_election: true
I20260927 11:13:18.039216 4889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90ce159e0ec40d999841c8498b464a4" candidate_uuid: "f3f809fa41cb4339b213eb49e393a915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d326d46da41e4cac82607b8d7290ea9d" is_pre_election: true
I20260927 11:13:18.039309 4756 raft_consensus.cc:2454] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3f809fa41cb4339b213eb49e393a915 in term 0.
I20260927 11:13:18.039322 4889 raft_consensus.cc:2454] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3f809fa41cb4339b213eb49e393a915 in term 0.
I20260927 11:13:18.039525 4556 leader_election.cc:302] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d326d46da41e4cac82607b8d7290ea9d, f3f809fa41cb4339b213eb49e393a915; no voters:
I20260927 11:13:18.039654 4956 raft_consensus.cc:2781] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:18.039716 4956 raft_consensus.cc:482] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:18.039746 4956 raft_consensus.cc:3037] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:18.040356 4956 raft_consensus.cc:504] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41745 } } peers { permanent_uuid: "f3f809fa41cb4339b213eb49e393a915" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38219 } } peers { permanent_uuid: "d326d46da41e4cac82607b8d7290ea9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41133 } }
I20260927 11:13:18.040472 4956 leader_election.cc:288] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [CANDIDATE]: Term 1 election: Requested vote from peers 9cfdaad9f7a94dbba3208ef70d7d89ea (127.0.98.2:41745), d326d46da41e4cac82607b8d7290ea9d (127.0.98.3:41133)
I20260927 11:13:18.040591 4756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90ce159e0ec40d999841c8498b464a4" candidate_uuid: "f3f809fa41cb4339b213eb49e393a915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea"
I20260927 11:13:18.040627 4889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f90ce159e0ec40d999841c8498b464a4" candidate_uuid: "f3f809fa41cb4339b213eb49e393a915" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d326d46da41e4cac82607b8d7290ea9d"
I20260927 11:13:18.040661 4756 raft_consensus.cc:3037] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:18.040684 4889 raft_consensus.cc:3037] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:18.041504 4889 raft_consensus.cc:2454] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3f809fa41cb4339b213eb49e393a915 in term 1.
I20260927 11:13:18.041504 4756 raft_consensus.cc:2454] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3f809fa41cb4339b213eb49e393a915 in term 1.
I20260927 11:13:18.041639 4556 leader_election.cc:302] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d326d46da41e4cac82607b8d7290ea9d, f3f809fa41cb4339b213eb49e393a915; no voters:
I20260927 11:13:18.041759 4956 raft_consensus.cc:2781] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:18.041865 4956 raft_consensus.cc:686] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [term 1 LEADER]: Becoming Leader. State: Replica: f3f809fa41cb4339b213eb49e393a915, State: Running, Role: LEADER
I20260927 11:13:18.041956 4956 consensus_queue.cc:237] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41745 } } peers { permanent_uuid: "f3f809fa41cb4339b213eb49e393a915" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38219 } } peers { permanent_uuid: "d326d46da41e4cac82607b8d7290ea9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41133 } }
I20260927 11:13:18.042680 4478 catalog_manager.cc:5495] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3f809fa41cb4339b213eb49e393a915 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f3f809fa41cb4339b213eb49e393a915" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3f809fa41cb4339b213eb49e393a915" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d326d46da41e4cac82607b8d7290ea9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41133 } health_report { overall_health: UNKNOWN } } }
W20260927 11:13:18.059979 4670 tablet.cc:2369] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:13:18.069860 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.067574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.0.98.1:38219)
0927 11:13:18.067640 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.067642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.067651 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:13:18.067715 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.067715 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.067732 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.067817 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:18.067952 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.068381 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.069074 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.069260 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.069329 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:18.069484 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:18.069573 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:18.069683 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:18.069733 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 11:13:18.069753 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:13:18.071373 4889 raft_consensus.cc:1261] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d [term 1 FOLLOWER]: Refusing update from remote peer f3f809fa41cb4339b213eb49e393a915: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:18.071404 4756 raft_consensus.cc:1261] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea [term 1 FOLLOWER]: Refusing update from remote peer f3f809fa41cb4339b213eb49e393a915: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:18.071696 4959 consensus_queue.cc:1035] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cfdaad9f7a94dbba3208ef70d7d89ea" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:18.071813 4956 consensus_queue.cc:1035] T f90ce159e0ec40d999841c8498b464a4 P f3f809fa41cb4339b213eb49e393a915 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d326d46da41e4cac82607b8d7290ea9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:18.074512 4964 mvcc.cc:205] Tried to move back new op lower bound from 7333919121698045952 to 7333919121580564480. Current Snapshot: MvccSnapshot[applied={T|T < 7333919121698045952}]
I20260927 11:13:18.076236 4478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:18.076319 4478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:18.077003 4478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36a88adc6627438e8e432461826bc3ec: Sending DeleteTablet for 3 replicas of tablet f90ce159e0ec40d999841c8498b464a4
I20260927 11:13:18.077294 4869 tablet_service.cc:1514] Processing DeleteTablet for tablet f90ce159e0ec40d999841c8498b464a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35138
I20260927 11:13:18.077342 4736 tablet_service.cc:1514] Processing DeleteTablet for tablet f90ce159e0ec40d999841c8498b464a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39822
I20260927 11:13:18.077473 4973 tablet_replica.cc:330] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d: stopping tablet replica
I20260927 11:13:18.077472 4603 tablet_service.cc:1514] Processing DeleteTablet for tablet f90ce159e0ec40d999841c8498b464a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32926
I20260927 11:13:18.077564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 4541
I20260927 11:13:18.077584 4973 raft_consensus.cc:2229] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:18.077672 4973 raft_consensus.cc:2258] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:18.079016 4974 tablet_replica.cc:330] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea: stopping tablet replica
I20260927 11:13:18.079105 4974 raft_consensus.cc:2229] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:18.079097 4973 ts_tablet_manager.cc:1905] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:18.079195 4974 raft_consensus.cc:2258] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:18.079545 4974 ts_tablet_manager.cc:1905] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:18.080596 4973 ts_tablet_manager.cc:1918] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:18.080662 4973 log.cc:1196] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/f90ce159e0ec40d999841c8498b464a4
I20260927 11:13:18.080919 4973 ts_tablet_manager.cc:1939] T f90ce159e0ec40d999841c8498b464a4 P d326d46da41e4cac82607b8d7290ea9d: Deleting consensus metadata
I20260927 11:13:18.081038 4974 ts_tablet_manager.cc:1918] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:18.081099 4974 log.cc:1196] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/f90ce159e0ec40d999841c8498b464a4
I20260927 11:13:18.081310 4463 catalog_manager.cc:4841] TS d326d46da41e4cac82607b8d7290ea9d (127.0.98.3:41133): tablet f90ce159e0ec40d999841c8498b464a4 (table test-table [id=2519bd77dbd740a19121722126f86b2c]) successfully deleted
I20260927 11:13:18.081370 4974 ts_tablet_manager.cc:1939] T f90ce159e0ec40d999841c8498b464a4 P 9cfdaad9f7a94dbba3208ef70d7d89ea: Deleting consensus metadata
I20260927 11:13:18.081702 4464 catalog_manager.cc:4841] TS 9cfdaad9f7a94dbba3208ef70d7d89ea (127.0.98.2:41745): tablet f90ce159e0ec40d999841c8498b464a4 (table test-table [id=2519bd77dbd740a19121722126f86b2c]) successfully deleted
W20260927 11:13:18.083290 4466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38219 (error 108)
I20260927 11:13:18.083642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 4674
I20260927 11:13:18.083722 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.083610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38219 (local address 127.0.0.1:32934)
0927 11:13:18.083645 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:18.083674 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260927 11:13:18.088331 4466 catalog_manager.cc:4584] TS f3f809fa41cb4339b213eb49e393a915 (127.0.98.1:38219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f90ce159e0ec40d999841c8498b464a4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38219: connect: Connection refused (error 111)
I20260927 11:13:18.088474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 4807
I20260927 11:13:18.092916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 4448
2026-09-27T11:13:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1964 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T11:13:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:18Z Disabled control of system clock
I20260927 11:13:18.119055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42263
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 11:13:18.195695 4980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:18.195871 4980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:18.195904 4980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:18.196969 4980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:18.197006 4980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:18.197018 4980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:18.197060 4980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:18.197088 4980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:18.198375 4980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42263
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:18.198607 4980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:18.198891 4980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:18.203470 4985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:18.203493 4986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:18.203565 4980 server_base.cc:1034] running on GCE node
W20260927 11:13:18.203516 4988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:18.203864 4980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:18.204129 4980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:18.205292 4980 hybrid_clock.cc:648] HybridClock initialized: now 1790507598205286 us; error 38 us; skew 500 ppm
I20260927 11:13:18.206458 4980 webserver.cc:460] Webserver started at http://127.0.98.62:45121/ using document root <none> and password file <none>
I20260927 11:13:18.206689 4980 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:18.206754 4980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:18.206893 4980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:18.207832 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "58526e4cfafd4855804bfd50fa39db9a"
format_stamp: "Formatted at 2026-09-27 11:13:18 on dist-test-slave-9hq9"
server_key: "6cdbd2ca4c323dbc39a36e50bd1b8649"
server_key_iv: "9daa1b21b488cbe9ff26f19d079a5d1f"
server_key_version: "encryptionkey@0"
I20260927 11:13:18.208158 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "58526e4cfafd4855804bfd50fa39db9a"
format_stamp: "Formatted at 2026-09-27 11:13:18 on dist-test-slave-9hq9"
server_key: "6cdbd2ca4c323dbc39a36e50bd1b8649"
server_key_iv: "9daa1b21b488cbe9ff26f19d079a5d1f"
server_key_version: "encryptionkey@0"
I20260927 11:13:18.209342 4980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:13:18.210003 4994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:18.210179 4980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:18.210273 4980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "58526e4cfafd4855804bfd50fa39db9a"
format_stamp: "Formatted at 2026-09-27 11:13:18 on dist-test-slave-9hq9"
server_key: "6cdbd2ca4c323dbc39a36e50bd1b8649"
server_key_iv: "9daa1b21b488cbe9ff26f19d079a5d1f"
server_key_version: "encryptionkey@0"
I20260927 11:13:18.210353 4980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:18.213409 4980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:18.213651 4980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:18.213781 4980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:18.217178 4980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42263
I20260927 11:13:18.217206 5046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42263 every 8 connection(s)
I20260927 11:13:18.217490 4980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:18.218093 5047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:18.220953 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a: Bootstrap starting.
I20260927 11:13:18.221495 5047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:18.221779 5047 log.cc:824] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:18.222399 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a: No bootstrap required, opened a new log
I20260927 11:13:18.223474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 4980
I20260927 11:13:18.223572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:18.223732 392 external_mini_cluster.cc:1428] Setting key 46f1f8e0661817961389447a9731ac63
I20260927 11:13:18.224035 5047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58526e4cfafd4855804bfd50fa39db9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42263 } }
I20260927 11:13:18.224210 5047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:18.224296 5047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58526e4cfafd4855804bfd50fa39db9a, State: Initialized, Role: FOLLOWER
I20260927 11:13:18.224424 5047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58526e4cfafd4855804bfd50fa39db9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42263 } }
I20260927 11:13:18.224498 5047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:18.224519 5047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:18.224587 5047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:18.225209 5047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58526e4cfafd4855804bfd50fa39db9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42263 } }
I20260927 11:13:18.225313 5047 leader_election.cc:302] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58526e4cfafd4855804bfd50fa39db9a; no voters:
I20260927 11:13:18.225481 5047 leader_election.cc:288] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:18.225561 5052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:18.225736 5052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [term 1 LEADER]: Becoming Leader. State: Replica: 58526e4cfafd4855804bfd50fa39db9a, State: Running, Role: LEADER
I20260927 11:13:18.225751 5047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:18.225865 5052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58526e4cfafd4855804bfd50fa39db9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42263 } }
I20260927 11:13:18.226446 5054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58526e4cfafd4855804bfd50fa39db9a. Latest consensus state: current_term: 1 leader_uuid: "58526e4cfafd4855804bfd50fa39db9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58526e4cfafd4855804bfd50fa39db9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42263 } } }
I20260927 11:13:18.226516 5054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:18.226835 5061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:18.226949 5053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58526e4cfafd4855804bfd50fa39db9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58526e4cfafd4855804bfd50fa39db9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42263 } } }
I20260927 11:13:18.227020 5053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:18.227284 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.224200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34502 (local address 127.0.98.62:42263)
0927 11:13:18.224406 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.224414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.224441 (+ 27us) server_negotiation.cc:407] Connection header received
0927 11:13:18.224564 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.224570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.224610 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.224731 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:18.225361 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:18.225364 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:18.225364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:18.225368 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:18.225410 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:18.225423 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:18.225482 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:18.225593 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 11:13:18.225648 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260927 11:13:18.227313 5061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:18.228744 5061 catalog_manager.cc:1347] Generated new cluster ID: dd3be41756b64281a10fb9228bba987b
I20260927 11:13:18.228804 5061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:13:18.233857 5061 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:18.233989 5061 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:18.242921 5061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a: Generated new TSK 0
I20260927 11:13:18.243084 5061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:13:18.244933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42263
--builtin_ntp_servers=127.0.98.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 11:13:18.321971 5071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:18.322233 5071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:18.322271 5071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:18.323374 5071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:18.323434 5071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:18.323477 5071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:18.324790 5071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:18.325032 5071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:18.325300 5071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:18.330339 5077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:18.330382 5079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:18.330457 5076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:18.330708 5071 server_base.cc:1034] running on GCE node
I20260927 11:13:18.330899 5071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:18.331148 5071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:18.332309 5071 hybrid_clock.cc:648] HybridClock initialized: now 1790507598332280 us; error 41 us; skew 500 ppm
I20260927 11:13:18.333662 5071 webserver.cc:460] Webserver started at http://127.0.98.1:41693/ using document root <none> and password file <none>
I20260927 11:13:18.333918 5071 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:18.333979 5071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:18.334148 5071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:18.335403 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "c8f0ba6d4c7a400ca5751b811af45927"
format_stamp: "Formatted at 2026-09-27 11:13:18 on dist-test-slave-9hq9"
server_key: "e85ea1a28f90bf43b83b734d3de53e49"
server_key_iv: "1f9cf96fea2d8aa235ed388e8bfa7fbd"
server_key_version: "encryptionkey@0"
I20260927 11:13:18.335764 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c8f0ba6d4c7a400ca5751b811af45927"
format_stamp: "Formatted at 2026-09-27 11:13:18 on dist-test-slave-9hq9"
server_key: "e85ea1a28f90bf43b83b734d3de53e49"
server_key_iv: "1f9cf96fea2d8aa235ed388e8bfa7fbd"
server_key_version: "encryptionkey@0"
I20260927 11:13:18.337244 5071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:13:18.338027 5085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:18.338234 5071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:18.338315 5071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "c8f0ba6d4c7a400ca5751b811af45927"
format_stamp: "Formatted at 2026-09-27 11:13:18 on dist-test-slave-9hq9"
server_key: "e85ea1a28f90bf43b83b734d3de53e49"
server_key_iv: "1f9cf96fea2d8aa235ed388e8bfa7fbd"
server_key_version: "encryptionkey@0"
I20260927 11:13:18.338377 5071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:18.342741 5071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:18.342996 5071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:18.343124 5071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:18.343329 5071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:18.343690 5071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:18.343742 5071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:18.343775 5071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:18.343799 5071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:18.350646 5071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39867
I20260927 11:13:18.350673 5198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39867 every 8 connection(s)
I20260927 11:13:18.350972 5071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:13:18.353658 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.351577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54575 (local address 127.0.98.62:42263)
0927 11:13:18.351635 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.351637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.351867 (+ 230us) server_negotiation.cc:407] Connection header received
0927 11:13:18.352269 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.352270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.352276 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.352350 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:18.353448 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:18.353449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:18.353449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:18.353452 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:18.353465 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:18.353469 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:18.353527 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:18.353576 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 11:13:18.353589 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 11:13:18.353863 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.351555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42263 (local address 127.0.98.1:54575)
0927 11:13:18.351726 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:18.351763 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:18.352191 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:18.352397 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:18.352400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:18.352407 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:18.353324 (+ 917us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:13:18.353325 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:18.353350 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:13:18.353376 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:18.353525 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:18.353527 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:18.353529 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:13:18.353573 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 11:13:18.353617 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
I20260927 11:13:18.354315 5199 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42263
I20260927 11:13:18.354398 5199 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:18.354539 5199 heartbeater.cc:511] Master 127.0.98.62:42263 requested a full tablet report, sending...
I20260927 11:13:18.355052 5011 ts_manager.cc:194] Registered new tserver with Master: c8f0ba6d4c7a400ca5751b811af45927 (127.0.98.1:39867)
I20260927 11:13:18.359287 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 5071
I20260927 11:13:18.359385 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:13:18.359582 392 external_mini_cluster.cc:1428] Setting key c2748b88a5ba95699211596717cf1463
I20260927 11:13:18.360392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42263
--builtin_ntp_servers=127.0.98.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 11:13:18.439337 5202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:18.439522 5202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:18.439543 5202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:18.440640 5202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:18.440675 5202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:18.440698 5202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:18.442095 5202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:18.442313 5202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:18.442574 5202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:18.447593 5207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:18.447664 5202 server_base.cc:1034] running on GCE node
W20260927 11:13:18.447758 5208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:18.447804 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:18.448024 5202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:18.448267 5202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:18.449421 5202 hybrid_clock.cc:648] HybridClock initialized: now 1790507598449413 us; error 45 us; skew 500 ppm
I20260927 11:13:18.450680 5202 webserver.cc:460] Webserver started at http://127.0.98.2:36821/ using document root <none> and password file <none>
I20260927 11:13:18.450908 5202 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:18.450978 5202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:18.451129 5202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:18.452116 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "c69250911a694d6e8f273028d7d9241b"
format_stamp: "Formatted at 2026-09-27 11:13:18 on dist-test-slave-9hq9"
server_key: "70191e64dfdd501dda094f914add2c1f"
server_key_iv: "04caced764360e982f6f7fac5f1fba80"
server_key_version: "encryptionkey@0"
I20260927 11:13:18.452466 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c69250911a694d6e8f273028d7d9241b"
format_stamp: "Formatted at 2026-09-27 11:13:18 on dist-test-slave-9hq9"
server_key: "70191e64dfdd501dda094f914add2c1f"
server_key_iv: "04caced764360e982f6f7fac5f1fba80"
server_key_version: "encryptionkey@0"
I20260927 11:13:18.453680 5202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:13:18.454425 5216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:18.454592 5202 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:18.454684 5202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "c69250911a694d6e8f273028d7d9241b"
format_stamp: "Formatted at 2026-09-27 11:13:18 on dist-test-slave-9hq9"
server_key: "70191e64dfdd501dda094f914add2c1f"
server_key_iv: "04caced764360e982f6f7fac5f1fba80"
server_key_version: "encryptionkey@0"
I20260927 11:13:18.454759 5202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:18.457901 5202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:18.458184 5202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:18.458314 5202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:18.458524 5202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:18.458806 5202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:18.458858 5202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:18.458894 5202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:18.458922 5202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:18.464371 5202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37587
I20260927 11:13:18.464560 5329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37587 every 8 connection(s)
I20260927 11:13:18.464711 5202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:13:18.467376 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.465340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53475 (local address 127.0.98.62:42263)
0927 11:13:18.465382 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.465388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.465606 (+ 218us) server_negotiation.cc:407] Connection header received
0927 11:13:18.465998 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.465999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.466003 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.466117 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:18.467216 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:18.467217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:18.467217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:18.467219 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:18.467230 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:18.467233 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:18.467274 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:18.467319 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 11:13:18.467329 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:13:18.467583 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.465321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42263 (local address 127.0.98.2:53475)
0927 11:13:18.465483 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:18.465517 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:18.465920 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:18.466179 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:18.466183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:18.466190 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:18.467097 (+ 907us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:13:18.467098 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:18.467122 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:13:18.467146 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:18.467273 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:18.467276 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:18.467278 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:13:18.467326 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 11:13:18.467368 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260927 11:13:18.468056 5330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42263
I20260927 11:13:18.468150 5330 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:18.468288 5330 heartbeater.cc:511] Master 127.0.98.62:42263 requested a full tablet report, sending...
I20260927 11:13:18.468626 5011 ts_manager.cc:194] Registered new tserver with Master: c69250911a694d6e8f273028d7d9241b (127.0.98.2:37587)
I20260927 11:13:18.474601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 5202
I20260927 11:13:18.474697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:13:18.474893 392 external_mini_cluster.cc:1428] Setting key 5a33344ef5f77a37f02365bb60f70635
I20260927 11:13:18.475709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42263
--builtin_ntp_servers=127.0.98.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 11:13:18.551776 5333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:18.551950 5333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:18.551970 5333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:18.553009 5333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:18.553040 5333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:18.553063 5333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:18.554345 5333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:18.554513 5333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:18.554730 5333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:18.559433 5339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:18.559492 5341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:18.559494 5338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:18.559648 5333 server_base.cc:1034] running on GCE node
I20260927 11:13:18.559873 5333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:18.560045 5333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:18.561179 5333 hybrid_clock.cc:648] HybridClock initialized: now 1790507598561171 us; error 26 us; skew 500 ppm
I20260927 11:13:18.562436 5333 webserver.cc:460] Webserver started at http://127.0.98.3:43311/ using document root <none> and password file <none>
I20260927 11:13:18.562669 5333 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:18.562716 5333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:18.562875 5333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:18.563848 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "535f92b0595c4f7ab7f4fc10679659d9"
format_stamp: "Formatted at 2026-09-27 11:13:18 on dist-test-slave-9hq9"
server_key: "61d5d109a1ef947e428f12c4a2c8ba5d"
server_key_iv: "c92e30899584e885201784a888d80f31"
server_key_version: "encryptionkey@0"
I20260927 11:13:18.564196 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "535f92b0595c4f7ab7f4fc10679659d9"
format_stamp: "Formatted at 2026-09-27 11:13:18 on dist-test-slave-9hq9"
server_key: "61d5d109a1ef947e428f12c4a2c8ba5d"
server_key_iv: "c92e30899584e885201784a888d80f31"
server_key_version: "encryptionkey@0"
I20260927 11:13:18.565390 5333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:13:18.566090 5347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:18.566241 5333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:18.566306 5333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "535f92b0595c4f7ab7f4fc10679659d9"
format_stamp: "Formatted at 2026-09-27 11:13:18 on dist-test-slave-9hq9"
server_key: "61d5d109a1ef947e428f12c4a2c8ba5d"
server_key_iv: "c92e30899584e885201784a888d80f31"
server_key_version: "encryptionkey@0"
I20260927 11:13:18.566390 5333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:18.569463 5333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:18.569705 5333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:18.569828 5333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:18.570050 5333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:18.570385 5333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:18.570416 5333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:18.570467 5333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:18.570497 5333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:18.575789 5333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36287
I20260927 11:13:18.575829 5460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36287 every 8 connection(s)
I20260927 11:13:18.576108 5333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:13:18.578822 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.576739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44259 (local address 127.0.98.62:42263)
0927 11:13:18.576792 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.576794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.576990 (+ 196us) server_negotiation.cc:407] Connection header received
0927 11:13:18.577443 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.577445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.577450 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.577492 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:18.578569 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:18.578571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:18.578571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:18.578576 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:18.578590 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:18.578594 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:18.578652 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:18.578739 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 11:13:18.578755 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 11:13:18.578991 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.576711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42263 (local address 127.0.98.3:44259)
0927 11:13:18.576889 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:18.576916 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:18.577345 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:18.577559 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:18.577563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:18.577571 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:18.578415 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:13:18.578416 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:18.578439 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:13:18.578463 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:18.578650 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:18.578654 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:18.578657 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:18.578747 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 11:13:18.578799 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260927 11:13:18.579532 5461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42263
I20260927 11:13:18.579630 5461 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:18.579777 5461 heartbeater.cc:511] Master 127.0.98.62:42263 requested a full tablet report, sending...
I20260927 11:13:18.580011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 5333
I20260927 11:13:18.580114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:13:18.580116 5011 ts_manager.cc:194] Registered new tserver with Master: 535f92b0595c4f7ab7f4fc10679659d9 (127.0.98.3:36287)
I20260927 11:13:18.580305 392 external_mini_cluster.cc:1428] Setting key 4bfffb238bc5be5468a538ee88e29077
I20260927 11:13:18.581001 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:13:18.583189 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.582258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.0.98.62:42263)
0927 11:13:18.582290 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.582292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.582401 (+ 109us) server_negotiation.cc:407] Connection header received
0927 11:13:18.582450 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.582450 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.582453 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.582499 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:18.583058 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:18.583058 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:18.583058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:18.583061 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:18.583070 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:18.583073 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:18.583109 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:18.583141 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 11:13:18.583150 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 11:13:18.584476 5011 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:13:18.585826 5011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:18.589197 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.587678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39867 (local address 127.0.0.1:55948)
0927 11:13:18.587818 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:18.587853 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:18.587903 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:18.588221 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:18.588222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:18.588225 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:18.588840 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:13:18.588841 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:18.588847 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:13:18.588859 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:18.589051 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:18.589056 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:18.589059 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:18.589118 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 11:13:18.589131 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":48,"threads_started":1}
I20260927 11:13:18.589351 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.587685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37587 (local address 127.0.0.1:45534)
0927 11:13:18.587833 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:18.587856 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:18.587905 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:18.588196 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:18.588201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:18.588207 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:18.588746 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:13:18.588746 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:18.588757 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:13:18.588772 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:18.589206 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:18.589207 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:18.589208 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 11:13:18.589269 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 11:13:18.589284 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260927 11:13:18.589488 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.587918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.0.98.1:39867)
0927 11:13:18.588061 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.588065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.588075 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:13:18.588100 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.588102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.588112 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.588404 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:18.588928 (+ 524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:18.588935 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:18.588936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:18.588940 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:18.588969 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:18.588991 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:18.589056 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:18.589374 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 11:13:18.589392 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260927 11:13:18.589579 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.587721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36287 (local address 127.0.0.1:59330)
0927 11:13:18.588066 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:18.588076 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:18.588117 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:18.588840 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:18.588840 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:18.588843 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:18.589218 (+ 375us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:13:18.589218 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:18.589224 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:13:18.589235 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:18.589452 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:18.589452 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:18.589453 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 11:13:18.589494 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 11:13:18.589506 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":72,"threads_started":1}
I20260927 11:13:18.589635 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.587896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45534 (local address 127.0.98.2:37587)
0927 11:13:18.588039 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.588043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.588054 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:13:18.588081 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.588083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.588094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.588196 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:18.588857 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:18.588865 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:18.588866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:18.588870 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:18.588899 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:18.588922 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:18.588983 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:18.589515 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 11:13:18.589530 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260927 11:13:18.589790 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.588098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59330 (local address 127.0.98.3:36287)
0927 11:13:18.588234 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.588238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.588246 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:13:18.588265 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.588267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.588274 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.588348 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:18.589319 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:18.589325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:18.589325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:18.589329 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:18.589358 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:18.589377 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:18.589428 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:18.589703 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 11:13:18.589718 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260927 11:13:18.590126 5133 tablet_service.cc:1467] Processing CreateTablet for tablet c3292260d6f746d498c89b7db0bc042b (DEFAULT_TABLE table=test-table [id=6c45ed6158714078857d0e75d70080ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:18.590226 5264 tablet_service.cc:1467] Processing CreateTablet for tablet c3292260d6f746d498c89b7db0bc042b (DEFAULT_TABLE table=test-table [id=6c45ed6158714078857d0e75d70080ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:18.590296 5395 tablet_service.cc:1467] Processing CreateTablet for tablet c3292260d6f746d498c89b7db0bc042b (DEFAULT_TABLE table=test-table [id=6c45ed6158714078857d0e75d70080ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:18.590469 5133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3292260d6f746d498c89b7db0bc042b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:18.590540 5395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3292260d6f746d498c89b7db0bc042b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:18.590566 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3292260d6f746d498c89b7db0bc042b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:18.592761 5475 tablet_bootstrap.cc:492] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b: Bootstrap starting.
I20260927 11:13:18.593377 5475 tablet_bootstrap.cc:654] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:18.593464 5476 tablet_bootstrap.cc:492] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927: Bootstrap starting.
I20260927 11:13:18.593674 5477 tablet_bootstrap.cc:492] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9: Bootstrap starting.
I20260927 11:13:18.593885 5475 log.cc:824] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:18.594132 5476 tablet_bootstrap.cc:654] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:18.594312 5477 tablet_bootstrap.cc:654] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:18.594452 5476 log.cc:824] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:18.594619 5477 log.cc:824] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:18.594735 5475 tablet_bootstrap.cc:492] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b: No bootstrap required, opened a new log
I20260927 11:13:18.594798 5475 ts_tablet_manager.cc:1397] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:18.595172 5476 tablet_bootstrap.cc:492] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927: No bootstrap required, opened a new log
I20260927 11:13:18.595239 5476 ts_tablet_manager.cc:1397] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 11:13:18.597339 5475 raft_consensus.cc:348] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f0ba6d4c7a400ca5751b811af45927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39867 } } peers { permanent_uuid: "c69250911a694d6e8f273028d7d9241b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37587 } } peers { permanent_uuid: "535f92b0595c4f7ab7f4fc10679659d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36287 } }
I20260927 11:13:18.597510 5475 raft_consensus.cc:374] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:18.597544 5475 raft_consensus.cc:729] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c69250911a694d6e8f273028d7d9241b, State: Initialized, Role: FOLLOWER
I20260927 11:13:18.597668 5475 consensus_queue.cc:260] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f0ba6d4c7a400ca5751b811af45927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39867 } } peers { permanent_uuid: "c69250911a694d6e8f273028d7d9241b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37587 } } peers { permanent_uuid: "535f92b0595c4f7ab7f4fc10679659d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36287 } }
I20260927 11:13:18.597687 5476 raft_consensus.cc:348] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f0ba6d4c7a400ca5751b811af45927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39867 } } peers { permanent_uuid: "c69250911a694d6e8f273028d7d9241b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37587 } } peers { permanent_uuid: "535f92b0595c4f7ab7f4fc10679659d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36287 } }
I20260927 11:13:18.597838 5476 raft_consensus.cc:374] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:18.597883 5476 raft_consensus.cc:729] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8f0ba6d4c7a400ca5751b811af45927, State: Initialized, Role: FOLLOWER
I20260927 11:13:18.597921 5475 ts_tablet_manager.cc:1428] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 11:13:18.597961 5330 heartbeater.cc:503] Master 127.0.98.62:42263 was elected leader, sending a full tablet report...
I20260927 11:13:18.598014 5476 consensus_queue.cc:260] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f0ba6d4c7a400ca5751b811af45927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39867 } } peers { permanent_uuid: "c69250911a694d6e8f273028d7d9241b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37587 } } peers { permanent_uuid: "535f92b0595c4f7ab7f4fc10679659d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36287 } }
I20260927 11:13:18.598306 5476 ts_tablet_manager.cc:1428] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 11:13:18.598330 5199 heartbeater.cc:503] Master 127.0.98.62:42263 was elected leader, sending a full tablet report...
I20260927 11:13:18.598594 5477 tablet_bootstrap.cc:492] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9: No bootstrap required, opened a new log
I20260927 11:13:18.598660 5477 ts_tablet_manager.cc:1397] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 11:13:18.600605 5477 raft_consensus.cc:348] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f0ba6d4c7a400ca5751b811af45927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39867 } } peers { permanent_uuid: "c69250911a694d6e8f273028d7d9241b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37587 } } peers { permanent_uuid: "535f92b0595c4f7ab7f4fc10679659d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36287 } }
I20260927 11:13:18.600736 5477 raft_consensus.cc:374] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:18.600780 5477 raft_consensus.cc:729] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 535f92b0595c4f7ab7f4fc10679659d9, State: Initialized, Role: FOLLOWER
I20260927 11:13:18.600886 5477 consensus_queue.cc:260] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f0ba6d4c7a400ca5751b811af45927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39867 } } peers { permanent_uuid: "c69250911a694d6e8f273028d7d9241b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37587 } } peers { permanent_uuid: "535f92b0595c4f7ab7f4fc10679659d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36287 } }
I20260927 11:13:18.601120 5477 ts_tablet_manager.cc:1428] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:13:18.601136 5461 heartbeater.cc:503] Master 127.0.98.62:42263 was elected leader, sending a full tablet report...
W20260927 11:13:18.601506 5200 tablet.cc:2369] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:13:18.647552 5482 raft_consensus.cc:482] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:18.647701 5482 raft_consensus.cc:504] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f0ba6d4c7a400ca5751b811af45927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39867 } } peers { permanent_uuid: "c69250911a694d6e8f273028d7d9241b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37587 } } peers { permanent_uuid: "535f92b0595c4f7ab7f4fc10679659d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36287 } }
I20260927 11:13:18.648077 5482 leader_election.cc:288] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c69250911a694d6e8f273028d7d9241b (127.0.98.2:37587), 535f92b0595c4f7ab7f4fc10679659d9 (127.0.98.3:36287)
I20260927 11:13:18.649348 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.648275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60847 (local address 127.0.98.2:37587)
0927 11:13:18.648364 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.648366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.648374 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:13:18.648399 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.648399 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.648404 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.648460 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:18.649080 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:18.649081 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:18.649081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:18.649085 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:18.649107 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:18.649126 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:18.649187 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:18.649274 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 11:13:18.649289 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 11:13:18.649348 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.648225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37587 (local address 127.0.98.1:60847)
0927 11:13:18.648276 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:18.648288 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:18.648323 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:18.648479 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:18.648479 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:18.648482 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:18.648959 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:13:18.648959 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:18.648967 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:13:18.648980 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:18.649196 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:18.649197 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:18.649198 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 11:13:18.649268 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 11:13:18.649283 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 11:13:18.649456 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.648302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55637 (local address 127.0.98.3:36287)
0927 11:13:18.648349 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.648352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.648452 (+ 100us) server_negotiation.cc:407] Connection header received
0927 11:13:18.648548 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.648548 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.648554 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.648615 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:18.649247 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:18.649248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:18.649249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:18.649253 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:18.649267 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:18.649270 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:18.649325 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:18.649380 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 11:13:18.649393 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:13:18.649467 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.648236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36287 (local address 127.0.98.1:55637)
0927 11:13:18.648380 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:18.648388 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:18.648475 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:18.648619 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:18.648619 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:18.648622 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:18.649172 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:13:18.649173 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:18.649179 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:13:18.649190 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:18.649325 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:18.649326 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:18.649327 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 11:13:18.649378 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 11:13:18.649392 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260927 11:13:18.649681 5284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3292260d6f746d498c89b7db0bc042b" candidate_uuid: "c8f0ba6d4c7a400ca5751b811af45927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c69250911a694d6e8f273028d7d9241b" is_pre_election: true
I20260927 11:13:18.649811 5284 raft_consensus.cc:2454] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f0ba6d4c7a400ca5751b811af45927 in term 0.
I20260927 11:13:18.649755 5415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3292260d6f746d498c89b7db0bc042b" candidate_uuid: "c8f0ba6d4c7a400ca5751b811af45927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "535f92b0595c4f7ab7f4fc10679659d9" is_pre_election: true
I20260927 11:13:18.649904 5415 raft_consensus.cc:2454] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8f0ba6d4c7a400ca5751b811af45927 in term 0.
I20260927 11:13:18.650027 5089 leader_election.cc:302] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c69250911a694d6e8f273028d7d9241b, c8f0ba6d4c7a400ca5751b811af45927; no voters:
I20260927 11:13:18.650179 5482 raft_consensus.cc:2781] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:18.650230 5482 raft_consensus.cc:482] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:18.650281 5482 raft_consensus.cc:3037] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:18.650942 5482 raft_consensus.cc:504] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f0ba6d4c7a400ca5751b811af45927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39867 } } peers { permanent_uuid: "c69250911a694d6e8f273028d7d9241b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37587 } } peers { permanent_uuid: "535f92b0595c4f7ab7f4fc10679659d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36287 } }
I20260927 11:13:18.651075 5482 leader_election.cc:288] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [CANDIDATE]: Term 1 election: Requested vote from peers c69250911a694d6e8f273028d7d9241b (127.0.98.2:37587), 535f92b0595c4f7ab7f4fc10679659d9 (127.0.98.3:36287)
I20260927 11:13:18.651201 5284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3292260d6f746d498c89b7db0bc042b" candidate_uuid: "c8f0ba6d4c7a400ca5751b811af45927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c69250911a694d6e8f273028d7d9241b"
I20260927 11:13:18.651223 5415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3292260d6f746d498c89b7db0bc042b" candidate_uuid: "c8f0ba6d4c7a400ca5751b811af45927" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "535f92b0595c4f7ab7f4fc10679659d9"
I20260927 11:13:18.651271 5284 raft_consensus.cc:3037] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:18.651288 5415 raft_consensus.cc:3037] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:18.651901 5284 raft_consensus.cc:2454] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f0ba6d4c7a400ca5751b811af45927 in term 1.
I20260927 11:13:18.652019 5415 raft_consensus.cc:2454] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8f0ba6d4c7a400ca5751b811af45927 in term 1.
I20260927 11:13:18.652108 5089 leader_election.cc:302] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c69250911a694d6e8f273028d7d9241b, c8f0ba6d4c7a400ca5751b811af45927; no voters:
I20260927 11:13:18.652202 5482 raft_consensus.cc:2781] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:18.652364 5482 raft_consensus.cc:686] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [term 1 LEADER]: Becoming Leader. State: Replica: c8f0ba6d4c7a400ca5751b811af45927, State: Running, Role: LEADER
I20260927 11:13:18.652431 5482 consensus_queue.cc:237] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f0ba6d4c7a400ca5751b811af45927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39867 } } peers { permanent_uuid: "c69250911a694d6e8f273028d7d9241b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37587 } } peers { permanent_uuid: "535f92b0595c4f7ab7f4fc10679659d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36287 } }
I20260927 11:13:18.653158 5011 catalog_manager.cc:5495] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8f0ba6d4c7a400ca5751b811af45927 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c8f0ba6d4c7a400ca5751b811af45927" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8f0ba6d4c7a400ca5751b811af45927" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c69250911a694d6e8f273028d7d9241b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "535f92b0595c4f7ab7f4fc10679659d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36287 } health_report { overall_health: UNKNOWN } } }
I20260927 11:13:18.665109 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.664267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55964 (local address 127.0.98.1:39867)
0927 11:13:18.664306 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.664308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.664317 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:13:18.664369 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.664370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.664380 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.664429 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:18.664907 (+ 478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:18.664908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:18.664908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:18.664912 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:18.664924 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:18.664928 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:18.664974 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:18.665032 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 11:13:18.665045 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:13:18.666692 5284 raft_consensus.cc:1261] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b [term 1 FOLLOWER]: Refusing update from remote peer c8f0ba6d4c7a400ca5751b811af45927: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:18.666703 5415 raft_consensus.cc:1261] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9 [term 1 FOLLOWER]: Refusing update from remote peer c8f0ba6d4c7a400ca5751b811af45927: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:18.666973 5485 consensus_queue.cc:1035] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "535f92b0595c4f7ab7f4fc10679659d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:18.667093 5482 consensus_queue.cc:1035] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c69250911a694d6e8f273028d7d9241b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:18.669211 5488 mvcc.cc:205] Tried to move back new op lower bound from 7333919124136493056 to 7333919124081012736. Current Snapshot: MvccSnapshot[applied={T|T < 7333919124136493056}]
I20260927 11:13:18.669713 5489 mvcc.cc:205] Tried to move back new op lower bound from 7333919124136493056 to 7333919124081012736. Current Snapshot: MvccSnapshot[applied={T|T < 7333919124136493056}]
I20260927 11:13:18.670096 5490 mvcc.cc:205] Tried to move back new op lower bound from 7333919124136493056 to 7333919124081012736. Current Snapshot: MvccSnapshot[applied={T|T < 7333919124136493056}]
I20260927 11:13:18.671967 5010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:18.672055 5010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:18.672758 5010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58526e4cfafd4855804bfd50fa39db9a: Sending DeleteTablet for 3 replicas of tablet c3292260d6f746d498c89b7db0bc042b
I20260927 11:13:18.673084 5395 tablet_service.cc:1514] Processing DeleteTablet for tablet c3292260d6f746d498c89b7db0bc042b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:18 UTC) from {username='slave'} at 127.0.0.1:59330
I20260927 11:13:18.673180 5133 tablet_service.cc:1514] Processing DeleteTablet for tablet c3292260d6f746d498c89b7db0bc042b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:18 UTC) from {username='slave'} at 127.0.0.1:55948
I20260927 11:13:18.673309 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet c3292260d6f746d498c89b7db0bc042b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:18 UTC) from {username='slave'} at 127.0.0.1:45534
I20260927 11:13:18.673347 5498 tablet_replica.cc:330] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9: stopping tablet replica
I20260927 11:13:18.673508 5498 raft_consensus.cc:2229] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:18.673580 5499 tablet_replica.cc:330] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927: stopping tablet replica
I20260927 11:13:18.673671 5499 raft_consensus.cc:2229] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:13:18.673671 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 5071
I20260927 11:13:18.673784 5499 raft_consensus.cc:2258] T c3292260d6f746d498c89b7db0bc042b P c8f0ba6d4c7a400ca5751b811af45927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:18.674306 5498 raft_consensus.cc:2258] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:18.674513 5500 tablet_replica.cc:330] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b: stopping tablet replica
I20260927 11:13:18.674607 5500 raft_consensus.cc:2229] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:18.674705 5500 raft_consensus.cc:2258] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:18.675001 5498 ts_tablet_manager.cc:1905] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:18.675990 5500 ts_tablet_manager.cc:1905] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:18.676296 5498 ts_tablet_manager.cc:1918] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:18.676378 5498 log.cc:1196] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/c3292260d6f746d498c89b7db0bc042b
I20260927 11:13:18.676604 5498 ts_tablet_manager.cc:1939] T c3292260d6f746d498c89b7db0bc042b P 535f92b0595c4f7ab7f4fc10679659d9: Deleting consensus metadata
I20260927 11:13:18.677230 4997 catalog_manager.cc:4841] TS 535f92b0595c4f7ab7f4fc10679659d9 (127.0.98.3:36287): tablet c3292260d6f746d498c89b7db0bc042b (table test-table [id=6c45ed6158714078857d0e75d70080ec]) successfully deleted
I20260927 11:13:18.677636 5500 ts_tablet_manager.cc:1918] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:18.677691 5500 log.cc:1196] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/c3292260d6f746d498c89b7db0bc042b
I20260927 11:13:18.677928 5500 ts_tablet_manager.cc:1939] T c3292260d6f746d498c89b7db0bc042b P c69250911a694d6e8f273028d7d9241b: Deleting consensus metadata
I20260927 11:13:18.678320 4998 catalog_manager.cc:4841] TS c69250911a694d6e8f273028d7d9241b (127.0.98.2:37587): tablet c3292260d6f746d498c89b7db0bc042b (table test-table [id=6c45ed6158714078857d0e75d70080ec]) successfully deleted
W20260927 11:13:18.680145 4995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39867 (error 108)
I20260927 11:13:18.680619 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.680518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39867 (local address 127.0.0.1:55968)
0927 11:13:18.680563 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:18.680596 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 11:13:18.680701 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 5202
W20260927 11:13:18.680766 4995 catalog_manager.cc:4584] TS c8f0ba6d4c7a400ca5751b811af45927 (127.0.98.1:39867): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3292260d6f746d498c89b7db0bc042b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39867: connect: Connection refused (error 111)
I20260927 11:13:18.685027 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 5333
I20260927 11:13:18.689355 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 4980
2026-09-27T11:13:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (596 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 11:13:18.709252 392 mini_oidc.cc:152] Starting JWKS server
I20260927 11:13:18.710318 392 webserver.cc:460] Webserver started at https://127.0.0.1:33231/ using document root <none> and password file <none>
I20260927 11:13:18.710433 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 11:13:18.710609 392 webserver.cc:460] Webserver started at http://127.0.0.1:43099/ using document root <none> and password file <none>
2026-09-27T11:13:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:18Z Disabled control of system clock
I20260927 11:13:18.717751 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43737
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43737
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33231/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/testchainca.cert with env {}
W20260927 11:13:18.797645 5507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:18.797817 5507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:18.797837 5507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:18.798929 5507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:18.798969 5507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:18.798981 5507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:18.798993 5507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:18.799005 5507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 11:13:18.799017 5507 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 11:13:18.799028 5507 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33231/jwks.json
I20260927 11:13:18.800426 5507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43737
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33231/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:18.800594 5507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:18.800735 5507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:18.805305 5515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:18.805320 5512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:18.805336 5513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:18.805421 5507 server_base.cc:1034] running on GCE node
I20260927 11:13:18.805809 5507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:18.806177 5507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:18.807333 5507 hybrid_clock.cc:648] HybridClock initialized: now 1790507598807324 us; error 38 us; skew 500 ppm
I20260927 11:13:18.808638 5507 webserver.cc:460] Webserver started at http://127.0.98.62:37895/ using document root <none> and password file <none>
I20260927 11:13:18.808897 5507 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:18.808974 5507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:18.809098 5507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:18.810189 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "2d45f67f0fc24f92a8cf3d7eea3f8400"
format_stamp: "Formatted at 2026-09-27 11:13:18 on dist-test-slave-9hq9"
server_key: "a4025b3f59e3f70c5a99e712f8374077"
server_key_iv: "c4a2752c5c093851d82e4ba4891884f7"
server_key_version: "encryptionkey@0"
I20260927 11:13:18.810559 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "2d45f67f0fc24f92a8cf3d7eea3f8400"
format_stamp: "Formatted at 2026-09-27 11:13:18 on dist-test-slave-9hq9"
server_key: "a4025b3f59e3f70c5a99e712f8374077"
server_key_iv: "c4a2752c5c093851d82e4ba4891884f7"
server_key_version: "encryptionkey@0"
I20260927 11:13:18.811991 5507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:13:18.812700 5521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:18.812865 5507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:18.812960 5507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "2d45f67f0fc24f92a8cf3d7eea3f8400"
format_stamp: "Formatted at 2026-09-27 11:13:18 on dist-test-slave-9hq9"
server_key: "a4025b3f59e3f70c5a99e712f8374077"
server_key_iv: "c4a2752c5c093851d82e4ba4891884f7"
server_key_version: "encryptionkey@0"
I20260927 11:13:18.813053 5507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:18.825672 5507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:18.825928 5507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:18.826126 5507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:18.829457 5507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43737
I20260927 11:13:18.829497 5573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43737 every 8 connection(s)
I20260927 11:13:18.829787 5507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:18.830644 5574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:18.831539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 5507
I20260927 11:13:18.831640 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:18.831813 392 external_mini_cluster.cc:1428] Setting key 8e28711573c9dd2670b3cd38d21d6a5d
I20260927 11:13:18.833173 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400: Bootstrap starting.
I20260927 11:13:18.833837 5574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:18.834162 5574 log.cc:824] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:18.834821 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400: No bootstrap required, opened a new log
I20260927 11:13:18.836766 5574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d45f67f0fc24f92a8cf3d7eea3f8400" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43737 } }
I20260927 11:13:18.836922 5574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:18.836956 5574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d45f67f0fc24f92a8cf3d7eea3f8400, State: Initialized, Role: FOLLOWER
I20260927 11:13:18.837100 5574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d45f67f0fc24f92a8cf3d7eea3f8400" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43737 } }
I20260927 11:13:18.837203 5574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:18.837240 5574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:18.837276 5574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:18.837425 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.832274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.0.98.62:43737)
0927 11:13:18.832467 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.832475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.832506 (+ 31us) server_negotiation.cc:407] Connection header received
0927 11:13:18.832992 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.833000 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.833073 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.833218 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:18.833419 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.834749 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.835495 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.836081 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.836211 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:18.836738 (+ 527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:18.836741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:18.836742 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:18.836922 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:18.836962 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:18.836976 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:18.837036 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:18.837163 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 11:13:18.837236 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260927 11:13:18.837787 5574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d45f67f0fc24f92a8cf3d7eea3f8400" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43737 } }
I20260927 11:13:18.837895 5574 leader_election.cc:302] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d45f67f0fc24f92a8cf3d7eea3f8400; no voters:
I20260927 11:13:18.838129 5574 leader_election.cc:288] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:18.838146 5579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:18.838249 5579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [term 1 LEADER]: Becoming Leader. State: Replica: 2d45f67f0fc24f92a8cf3d7eea3f8400, State: Running, Role: LEADER
I20260927 11:13:18.838253 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:13:18.838325 5579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d45f67f0fc24f92a8cf3d7eea3f8400" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43737 } }
I20260927 11:13:18.838395 5574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:18.838650 5579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d45f67f0fc24f92a8cf3d7eea3f8400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d45f67f0fc24f92a8cf3d7eea3f8400" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43737 } } }
I20260927 11:13:18.838714 5579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:18.838830 5579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d45f67f0fc24f92a8cf3d7eea3f8400. Latest consensus state: current_term: 1 leader_uuid: "2d45f67f0fc24f92a8cf3d7eea3f8400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d45f67f0fc24f92a8cf3d7eea3f8400" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43737 } } }
I20260927 11:13:18.838893 5579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:18.839470 5584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:18.839869 5584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:18.841197 5584 catalog_manager.cc:1347] Generated new cluster ID: dc3920622d9d4ae68da254e2f0d98926
I20260927 11:13:18.841246 5584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:13:18.846105 5584 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:18.846724 5584 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:18.850188 5584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d45f67f0fc24f92a8cf3d7eea3f8400: Generated new TSK 0
I20260927 11:13:18.850329 5584 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 11:13:18.876075 5604 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 11:13:18.876154 5604 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 11:13:18.876396 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:18.860950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35822 (local address 127.0.98.62:43737)
0927 11:13:18.861003 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:18.861005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:18.861130 (+ 125us) server_negotiation.cc:407] Connection header received
0927 11:13:18.861206 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:18.861207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:18.861218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:18.861283 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0927 11:13:18.861404 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.862431 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.863185 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.863513 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.863581 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:18.863740 (+ 159us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 11:13:18.876177 (+ 12437us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33231/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 11:13:18.876335 (+ 158us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35822: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33231/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 11:13:18.876456 5577 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35822: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33231/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 11:13:18.876721 5603 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:18.860877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43737 (local address 127.0.0.1:35822)
0927 11:13:18.861065 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:18.861074 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:18.861126 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:18.861294 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:18.861295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:18.861296 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 11:13:18.861345 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:18.861347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.862544 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.862544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:18.863111 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:18.863113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:18.863584 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:18.863587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:18.863630 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:18.863646 (+ 16us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 11:13:18.876328 (+ 12682us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33231/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 11:13:18.876406 (+ 78us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43737: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33231/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:43737: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33231/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 11:13:18.877296 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 5507
2026-09-27T11:13:18Z chronyd exiting
I20260927 11:13:18.912571 392 test_util.cc:175] -----------------------------------------------
I20260927 11:13:18.912635 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790507576023819-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (204 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 11:13:18.913326 392 mini_oidc.cc:152] Starting JWKS server
I20260927 11:13:18.913933 392 webserver.cc:460] Webserver started at https://127.0.0.1:39011/ using document root <none> and password file <none>
I20260927 11:13:18.913972 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 11:13:18.914167 392 webserver.cc:460] Webserver started at http://127.0.0.1:45225/ using document root <none> and password file <none>
2026-09-27T11:13:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:18Z Disabled control of system clock
I20260927 11:13:18.921288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38581
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38581
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39011/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/testchainca.cert with env {}
W20260927 11:13:18.996624 5610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:18.996856 5610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:18.996891 5610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:18.998039 5610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:18.998132 5610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:18.998157 5610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:18.998186 5610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:18.998209 5610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 11:13:18.998238 5610 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 11:13:18.998265 5610 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39011/jwks.json
I20260927 11:13:18.999711 5610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39011/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:18.999966 5610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:19.000175 5610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:19.004666 5618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:19.004813 5615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:19.004716 5616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:19.004907 5610 server_base.cc:1034] running on GCE node
I20260927 11:13:19.005131 5610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:19.005398 5610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:19.006546 5610 hybrid_clock.cc:648] HybridClock initialized: now 1790507599006544 us; error 37 us; skew 500 ppm
I20260927 11:13:19.007753 5610 webserver.cc:460] Webserver started at http://127.0.98.62:40015/ using document root <none> and password file <none>
I20260927 11:13:19.007973 5610 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:19.008044 5610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:19.008185 5610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:19.009142 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "1dd3b0272d3e4f19a28ea53b0b8c0eed"
format_stamp: "Formatted at 2026-09-27 11:13:19 on dist-test-slave-9hq9"
server_key: "51da55998c34919330233fe6029d873d"
server_key_iv: "0ba7b11f6283c9ce5697f9c1120ccbc4"
server_key_version: "encryptionkey@0"
I20260927 11:13:19.009496 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "1dd3b0272d3e4f19a28ea53b0b8c0eed"
format_stamp: "Formatted at 2026-09-27 11:13:19 on dist-test-slave-9hq9"
server_key: "51da55998c34919330233fe6029d873d"
server_key_iv: "0ba7b11f6283c9ce5697f9c1120ccbc4"
server_key_version: "encryptionkey@0"
I20260927 11:13:19.010776 5610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:13:19.011435 5624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:19.011592 5610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:19.011682 5610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "1dd3b0272d3e4f19a28ea53b0b8c0eed"
format_stamp: "Formatted at 2026-09-27 11:13:19 on dist-test-slave-9hq9"
server_key: "51da55998c34919330233fe6029d873d"
server_key_iv: "0ba7b11f6283c9ce5697f9c1120ccbc4"
server_key_version: "encryptionkey@0"
I20260927 11:13:19.011757 5610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:19.027133 5610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:19.027390 5610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:19.027537 5610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:19.031235 5610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38581
I20260927 11:13:19.031265 5676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38581 every 8 connection(s)
I20260927 11:13:19.031560 5610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:19.032374 5677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:19.034380 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed: Bootstrap starting.
I20260927 11:13:19.034778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 5610
I20260927 11:13:19.034876 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:19.034898 5677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:19.035055 392 external_mini_cluster.cc:1428] Setting key 7bf07fb3a61ebbb91a0915cc28b7ad17
I20260927 11:13:19.035188 5677 log.cc:824] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:19.035883 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed: No bootstrap required, opened a new log
I20260927 11:13:19.037909 5677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dd3b0272d3e4f19a28ea53b0b8c0eed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38581 } }
I20260927 11:13:19.038080 5677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:19.038125 5677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd3b0272d3e4f19a28ea53b0b8c0eed, State: Initialized, Role: FOLLOWER
I20260927 11:13:19.038235 5677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dd3b0272d3e4f19a28ea53b0b8c0eed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38581 } }
I20260927 11:13:19.038302 5677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:19.038331 5677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:19.038374 5677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:19.038889 5677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dd3b0272d3e4f19a28ea53b0b8c0eed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38581 } }
I20260927 11:13:19.038990 5677 leader_election.cc:302] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dd3b0272d3e4f19a28ea53b0b8c0eed; no voters:
I20260927 11:13:19.039142 5677 leader_election.cc:288] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:19.039280 5682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:19.039382 5682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [term 1 LEADER]: Becoming Leader. State: Replica: 1dd3b0272d3e4f19a28ea53b0b8c0eed, State: Running, Role: LEADER
I20260927 11:13:19.039402 5677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:19.039453 5682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dd3b0272d3e4f19a28ea53b0b8c0eed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38581 } }
I20260927 11:13:19.039779 5682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dd3b0272d3e4f19a28ea53b0b8c0eed. Latest consensus state: current_term: 1 leader_uuid: "1dd3b0272d3e4f19a28ea53b0b8c0eed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dd3b0272d3e4f19a28ea53b0b8c0eed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38581 } } }
I20260927 11:13:19.039788 5683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dd3b0272d3e4f19a28ea53b0b8c0eed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dd3b0272d3e4f19a28ea53b0b8c0eed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38581 } } }
I20260927 11:13:19.039866 5682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:19.039887 5683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:19.040599 5688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:19.041054 5688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:19.041554 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:19.035449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.0.98.62:38581)
0927 11:13:19.035634 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:19.035641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:19.035675 (+ 34us) server_negotiation.cc:407] Connection header received
0927 11:13:19.036138 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:19.036145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:19.036198 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:19.036300 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:19.036529 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.037336 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.038430 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.038815 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.038896 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:19.040554 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:19.040558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:19.040558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:19.040779 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:19.040818 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:19.040832 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:19.040902 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:19.040997 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 11:13:19.041179 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260927 11:13:19.042426 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:13:19.042506 5688 catalog_manager.cc:1347] Generated new cluster ID: f9708d7d9aa74d52ab168537e241673b
I20260927 11:13:19.042546 5688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:13:19.071456 5688 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:19.071908 5688 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:19.076817 5688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dd3b0272d3e4f19a28ea53b0b8c0eed: Generated new TSK 0
I20260927 11:13:19.076977 5688 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 11:13:19.108577 5707 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 11:13:19.108685 5707 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 11:13:19.108943 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:19.086824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.0.98.62:38581)
0927 11:13:19.086862 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:19.086864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:19.086953 (+ 89us) server_negotiation.cc:407] Connection header received
0927 11:13:19.087020 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:19.087021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:19.087030 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:19.087085 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0927 11:13:19.087231 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.087695 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.088848 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.089302 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.089379 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:19.089532 (+ 153us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 11:13:19.108655 (+ 19123us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 11:13:19.108869 (+ 214us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55700: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 11:13:19.109009 5681 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55700: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 11:13:19.109035 5706 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:19.086729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38581 (local address 127.0.0.1:55700)
0927 11:13:19.086879 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:19.086892 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:19.086957 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:19.087110 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:19.087111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:19.087112 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 11:13:19.087171 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:19.087173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.087791 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.087792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:19.088744 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:19.088746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.089382 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.089385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:19.089440 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:19.089445 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 11:13:19.108835 (+ 19390us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 11:13:19.108926 (+ 91us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38581: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39011/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260927 11:13:19.109225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 5610
2026-09-27T11:13:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (232 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 11:13:19.145892 392 mini_oidc.cc:152] Starting JWKS server
I20260927 11:13:19.146605 392 webserver.cc:460] Webserver started at https://127.0.0.1:35545/ using document root <none> and password file <none>
I20260927 11:13:19.146672 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 11:13:19.146821 392 webserver.cc:460] Webserver started at http://127.0.0.1:34915/ using document root <none> and password file <none>
2026-09-27T11:13:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:19Z Disabled control of system clock
I20260927 11:13:19.153683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33027
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33027
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35545/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 11:13:19.230131 5713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:19.230307 5713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:19.230329 5713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:19.231442 5713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:19.231487 5713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:19.231500 5713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:19.231542 5713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:19.231570 5713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 11:13:19.231592 5713 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 11:13:19.231617 5713 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35545/jwks.json
I20260927 11:13:19.232915 5713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33027
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35545/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:19.233147 5713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:19.233405 5713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:19.237931 5719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:19.238026 5718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:19.237959 5713 server_base.cc:1034] running on GCE node
W20260927 11:13:19.238113 5721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:19.238336 5713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:19.238621 5713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:19.239779 5713 hybrid_clock.cc:648] HybridClock initialized: now 1790507599239758 us; error 39 us; skew 500 ppm
I20260927 11:13:19.240949 5713 webserver.cc:460] Webserver started at http://127.0.98.62:33851/ using document root <none> and password file <none>
I20260927 11:13:19.241171 5713 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:19.241246 5713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:19.241389 5713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:19.242419 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "0a2d52fd7a1642b280df7380bd6269f3"
format_stamp: "Formatted at 2026-09-27 11:13:19 on dist-test-slave-9hq9"
server_key: "1d56cdbd650695340005b4bdd2e58f41"
server_key_iv: "086fc2a19a40c7873520321776b39b14"
server_key_version: "encryptionkey@0"
I20260927 11:13:19.242764 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a2d52fd7a1642b280df7380bd6269f3"
format_stamp: "Formatted at 2026-09-27 11:13:19 on dist-test-slave-9hq9"
server_key: "1d56cdbd650695340005b4bdd2e58f41"
server_key_iv: "086fc2a19a40c7873520321776b39b14"
server_key_version: "encryptionkey@0"
I20260927 11:13:19.243957 5713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:13:19.244589 5727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:19.244743 5713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:19.244835 5713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "0a2d52fd7a1642b280df7380bd6269f3"
format_stamp: "Formatted at 2026-09-27 11:13:19 on dist-test-slave-9hq9"
server_key: "1d56cdbd650695340005b4bdd2e58f41"
server_key_iv: "086fc2a19a40c7873520321776b39b14"
server_key_version: "encryptionkey@0"
I20260927 11:13:19.244907 5713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:19.267271 5713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:19.267545 5713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:19.267688 5713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:19.271514 5713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33027
I20260927 11:13:19.271546 5779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33027 every 8 connection(s)
I20260927 11:13:19.271863 5713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:19.272439 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:19.274901 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3: Bootstrap starting.
I20260927 11:13:19.275445 5780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:19.275727 5780 log.cc:824] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:19.276330 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3: No bootstrap required, opened a new log
I20260927 11:13:19.277768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 5713
I20260927 11:13:19.277868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:19.277824 5780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2d52fd7a1642b280df7380bd6269f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } }
I20260927 11:13:19.278023 5780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:19.278047 392 external_mini_cluster.cc:1428] Setting key 377ce7974f2cbf1e2a2f9e97f8cfa56b
I20260927 11:13:19.278101 5780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a2d52fd7a1642b280df7380bd6269f3, State: Initialized, Role: FOLLOWER
I20260927 11:13:19.278245 5780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2d52fd7a1642b280df7380bd6269f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } }
I20260927 11:13:19.278334 5780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:19.278370 5780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:19.278421 5780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:19.279000 5780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2d52fd7a1642b280df7380bd6269f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } }
I20260927 11:13:19.279134 5780 leader_election.cc:302] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a2d52fd7a1642b280df7380bd6269f3; no voters:
I20260927 11:13:19.279292 5780 leader_election.cc:288] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:19.279342 5785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:19.279500 5785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [term 1 LEADER]: Becoming Leader. State: Replica: 0a2d52fd7a1642b280df7380bd6269f3, State: Running, Role: LEADER
I20260927 11:13:19.279577 5780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:19.279608 5785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2d52fd7a1642b280df7380bd6269f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } }
I20260927 11:13:19.279976 5785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a2d52fd7a1642b280df7380bd6269f3. Latest consensus state: current_term: 1 leader_uuid: "0a2d52fd7a1642b280df7380bd6269f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2d52fd7a1642b280df7380bd6269f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } } }
I20260927 11:13:19.279990 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a2d52fd7a1642b280df7380bd6269f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a2d52fd7a1642b280df7380bd6269f3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33027 } } }
I20260927 11:13:19.280057 5785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:19.280102 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:19.280531 5791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:19.281036 5791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:19.282495 5791 catalog_manager.cc:1347] Generated new cluster ID: 34da5e636df94211bba51a8276d3201c
I20260927 11:13:19.282548 5791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:13:19.284505 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:19.278481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.0.98.62:33027)
0927 11:13:19.278756 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:19.278764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:19.278802 (+ 38us) server_negotiation.cc:407] Connection header received
0927 11:13:19.278896 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:19.278901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:19.278960 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:19.279070 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:19.279341 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.281350 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.282465 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.283052 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.283170 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:19.283851 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:19.283855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:19.283855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:19.284048 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:19.284087 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:19.284101 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:19.284172 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:19.284240 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 11:13:19.284310 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":203,"thread_start_us":168,"threads_started":1}
I20260927 11:13:19.285434 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:13:19.286101 5791 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:19.286731 5791 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:19.296221 5791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a2d52fd7a1642b280df7380bd6269f3: Generated new TSK 0
I20260927 11:13:19.296360 5791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:13:19.311419 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:19.308222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.0.98.62:33027)
0927 11:13:19.308272 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:19.308273 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:19.308391 (+ 118us) server_negotiation.cc:407] Connection header received
0927 11:13:19.308448 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:19.308449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:19.308467 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:19.308520 (+ 53us) server_negotiation.cc:226] Negotiated authn=JWT
0927 11:13:19.308653 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.309081 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.310133 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.310313 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.310362 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:19.310524 (+ 162us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 11:13:19.311206 (+ 682us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35545/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 11:13:19.311354 (+ 148us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40326: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35545/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":30}
W20260927 11:13:19.311491 5809 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:19.308168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33027 (local address 127.0.0.1:40326)
0927 11:13:19.308302 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:19.308311 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:19.308387 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:19.308540 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:19.308541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:19.308543 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 11:13:19.308593 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:19.308595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.309150 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.309152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:19.310043 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:19.310044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.310377 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.310380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:19.310437 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:19.310441 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 11:13:19.311308 (+ 867us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35545/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 11:13:19.311394 (+ 86us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33027: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35545/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
I20260927 11:13:19.311719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 5713
E20260927 11:13:19.322325 5810 webserver.cc:533] Webserver: sslize failed:
E20260927 11:13:19.322373 5810 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T11:13:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (202 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T11:13:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:19Z Disabled control of system clock
I20260927 11:13:19.355432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39989
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39989
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/testchainca.cert with env {}
W20260927 11:13:19.431663 5814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:19.431839 5814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:19.431859 5814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:19.432900 5814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:19.432935 5814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:19.432948 5814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:19.432960 5814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:19.432972 5814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 11:13:19.432984 5814 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 11:13:19.432996 5814 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 11:13:19.434420 5814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:19.434602 5814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:19.434756 5814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:19.439216 5820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:19.439234 5822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:19.439261 5819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:19.439427 5814 server_base.cc:1034] running on GCE node
I20260927 11:13:19.439643 5814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:19.439882 5814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:19.441017 5814 hybrid_clock.cc:648] HybridClock initialized: now 1790507599441004 us; error 27 us; skew 500 ppm
I20260927 11:13:19.442245 5814 webserver.cc:460] Webserver started at http://127.0.98.62:44267/ using document root <none> and password file <none>
I20260927 11:13:19.442476 5814 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:19.442526 5814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:19.442659 5814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:19.443648 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "87e9f98aa256455d912c3c2cd6da00aa"
format_stamp: "Formatted at 2026-09-27 11:13:19 on dist-test-slave-9hq9"
server_key: "87b27dd74fd591116685287edb390841"
server_key_iv: "4b4fa37ac882ff890533c27fae59dade"
server_key_version: "encryptionkey@0"
I20260927 11:13:19.443985 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "87e9f98aa256455d912c3c2cd6da00aa"
format_stamp: "Formatted at 2026-09-27 11:13:19 on dist-test-slave-9hq9"
server_key: "87b27dd74fd591116685287edb390841"
server_key_iv: "4b4fa37ac882ff890533c27fae59dade"
server_key_version: "encryptionkey@0"
I20260927 11:13:19.445192 5814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:13:19.445842 5828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:19.446009 5814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:19.446122 5814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "87e9f98aa256455d912c3c2cd6da00aa"
format_stamp: "Formatted at 2026-09-27 11:13:19 on dist-test-slave-9hq9"
server_key: "87b27dd74fd591116685287edb390841"
server_key_iv: "4b4fa37ac882ff890533c27fae59dade"
server_key_version: "encryptionkey@0"
I20260927 11:13:19.446198 5814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:19.456182 5814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:19.456470 5814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:19.456617 5814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:19.460412 5814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39989
I20260927 11:13:19.460444 5880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39989 every 8 connection(s)
I20260927 11:13:19.460758 5814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:19.461660 5881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:19.463912 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa: Bootstrap starting.
I20260927 11:13:19.464474 5881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:19.464762 5881 log.cc:824] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:19.465462 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa: No bootstrap required, opened a new log
I20260927 11:13:19.467020 5881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e9f98aa256455d912c3c2cd6da00aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39989 } }
I20260927 11:13:19.467147 5881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:19.467195 5881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e9f98aa256455d912c3c2cd6da00aa, State: Initialized, Role: FOLLOWER
I20260927 11:13:19.467381 5881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e9f98aa256455d912c3c2cd6da00aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39989 } }
I20260927 11:13:19.467468 5881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:19.467509 5881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:19.467555 5881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:19.468035 5881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e9f98aa256455d912c3c2cd6da00aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39989 } }
I20260927 11:13:19.468142 5881 leader_election.cc:302] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87e9f98aa256455d912c3c2cd6da00aa; no voters:
I20260927 11:13:19.468321 5881 leader_election.cc:288] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:19.468367 5884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:19.468505 5884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [term 1 LEADER]: Becoming Leader. State: Replica: 87e9f98aa256455d912c3c2cd6da00aa, State: Running, Role: LEADER
I20260927 11:13:19.468564 5881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:19.468600 5884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e9f98aa256455d912c3c2cd6da00aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39989 } }
I20260927 11:13:19.468923 5885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87e9f98aa256455d912c3c2cd6da00aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e9f98aa256455d912c3c2cd6da00aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39989 } } }
I20260927 11:13:19.469026 5886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87e9f98aa256455d912c3c2cd6da00aa. Latest consensus state: current_term: 1 leader_uuid: "87e9f98aa256455d912c3c2cd6da00aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87e9f98aa256455d912c3c2cd6da00aa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39989 } } }
I20260927 11:13:19.469090 5886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:19.469055 5885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:19.469468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 5814
I20260927 11:13:19.469468 5892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:19.469578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:19.469745 392 external_mini_cluster.cc:1428] Setting key ad9857fd65ffbb3b4caf0254f113226b
I20260927 11:13:19.469960 5892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:19.471779 5892 catalog_manager.cc:1347] Generated new cluster ID: 4743b52d5c1e44f2bab6c2775bf4dd33
I20260927 11:13:19.471830 5892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:13:19.475274 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:19.470312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.0.98.62:39989)
0927 11:13:19.470764 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:19.470773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:19.470890 (+ 117us) server_negotiation.cc:407] Connection header received
0927 11:13:19.471162 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:19.471171 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:19.471254 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:19.471438 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:19.471659 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.472605 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.473380 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.473787 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.473937 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:19.474619 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:19.474622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:19.474623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:19.474795 (+ 172us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:19.474825 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:19.474835 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:19.474918 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:19.474992 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 11:13:19.475070 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":84,"threads_started":1}
I20260927 11:13:19.476258 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:13:19.479005 5892 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:19.479534 5892 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:19.500095 5892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87e9f98aa256455d912c3c2cd6da00aa: Generated new TSK 0
I20260927 11:13:19.500247 5892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:13:19.948051 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:19.507459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.0.98.62:39989)
0927 11:13:19.507516 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:19.507518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:19.507611 (+ 93us) server_negotiation.cc:407] Connection header received
0927 11:13:19.507675 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:19.507675 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:19.507684 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:19.507737 (+ 53us) server_negotiation.cc:226] Negotiated authn=JWT
0927 11:13:19.507866 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.508327 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.509369 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.509549 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.509593 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:19.509795 (+ 202us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 11:13:19.947843 (+438048us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 11:13:19.947992 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36538: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 11:13:19.948143 5900 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36538: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 11:13:19.948247 5910 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:19.507399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39989 (local address 127.0.0.1:36538)
0927 11:13:19.507544 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:19.507554 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:19.507615 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:19.507753 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:19.507754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:19.507756 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 11:13:19.507808 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:19.507809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.508408 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.508409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:19.509293 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:19.509295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:19.509619 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:19.509623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:19.509685 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:19.509690 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 11:13:19.947984 (+438294us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 11:13:19.948103 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39989: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":102,"threads_started":1}
I20260927 11:13:19.948589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 5814
2026-09-27T11:13:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (617 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (83 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (82 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:13:20 dist-test-slave-9hq9 krb5kdc[5922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:13:20 dist-test-slave-9hq9 krb5kdc[5922](info): set up 2 sockets
Sep 27 11:13:20 dist-test-slave-9hq9 krb5kdc[5922](info): commencing operation
krb5kdc: starting...
W20260927 11:13:21.122233 392 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:13:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:13:21.229936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36211
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:21.305790 5935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:21.305966 5935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:21.305989 5935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:21.307050 5935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:21.307085 5935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:21.307098 5935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:21.307111 5935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:21.307122 5935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:21.308593 5935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:21.308820 5935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:21.309036 5935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:21.313515 5941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:21.313596 5940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:21.313668 5935 server_base.cc:1034] running on GCE node
W20260927 11:13:21.313516 5943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:21.313921 5935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:21.314208 5935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:21.315361 5935 hybrid_clock.cc:648] HybridClock initialized: now 1790507601315338 us; error 27 us; skew 500 ppm
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:21.322361 5935 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:13:21.322741 5935 webserver.cc:460] Webserver started at http://127.0.98.62:45451/ using document root <none> and password file <none>
I20260927 11:13:21.322960 5935 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:21.323021 5935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:21.323167 5935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:21.324124 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "4f7ad69b289b4836937a7cd6f8182fae"
format_stamp: "Formatted at 2026-09-27 11:13:21 on dist-test-slave-9hq9"
server_key: "1b14ba394f0452ab97a8e16571bb8394"
server_key_iv: "db24ed30d9e13fdabf667b9f75b48fdd"
server_key_version: "encryptionkey@0"
I20260927 11:13:21.324453 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "4f7ad69b289b4836937a7cd6f8182fae"
format_stamp: "Formatted at 2026-09-27 11:13:21 on dist-test-slave-9hq9"
server_key: "1b14ba394f0452ab97a8e16571bb8394"
server_key_iv: "db24ed30d9e13fdabf667b9f75b48fdd"
server_key_version: "encryptionkey@0"
I20260927 11:13:21.325626 5935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:13:21.326349 5950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:21.326514 5935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:21.326598 5935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "4f7ad69b289b4836937a7cd6f8182fae"
format_stamp: "Formatted at 2026-09-27 11:13:21 on dist-test-slave-9hq9"
server_key: "1b14ba394f0452ab97a8e16571bb8394"
server_key_iv: "db24ed30d9e13fdabf667b9f75b48fdd"
server_key_version: "encryptionkey@0"
I20260927 11:13:21.326649 5935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:21.351464 5935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:21.353035 5935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:21.353137 5935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:21.356837 5935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36211
I20260927 11:13:21.357087 6002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36211 every 8 connection(s)
I20260927 11:13:21.357154 5935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:21.357780 6003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:21.359805 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae: Bootstrap starting.
I20260927 11:13:21.360280 6003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:21.360505 6003 log.cc:824] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:21.361075 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae: No bootstrap required, opened a new log
I20260927 11:13:21.362604 6003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ad69b289b4836937a7cd6f8182fae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } }
I20260927 11:13:21.362727 6003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:21.362747 6003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7ad69b289b4836937a7cd6f8182fae, State: Initialized, Role: FOLLOWER
I20260927 11:13:21.362838 6003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ad69b289b4836937a7cd6f8182fae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } }
I20260927 11:13:21.362891 6003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:21.362915 6003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:21.362946 6003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:21.363670 6003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ad69b289b4836937a7cd6f8182fae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } }
I20260927 11:13:21.363749 6003 leader_election.cc:302] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f7ad69b289b4836937a7cd6f8182fae; no voters:
I20260927 11:13:21.363937 6003 leader_election.cc:288] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:21.363982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 5935
I20260927 11:13:21.363973 6006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:21.364099 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:21.364156 6006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [term 1 LEADER]: Becoming Leader. State: Replica: 4f7ad69b289b4836937a7cd6f8182fae, State: Running, Role: LEADER
I20260927 11:13:21.364194 6003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:21.364285 392 external_mini_cluster.cc:1428] Setting key 313e9013652e7881bd82cb4f5b91a9be
I20260927 11:13:21.364248 6006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ad69b289b4836937a7cd6f8182fae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } }
I20260927 11:13:21.364530 6006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7ad69b289b4836937a7cd6f8182fae. Latest consensus state: current_term: 1 leader_uuid: "4f7ad69b289b4836937a7cd6f8182fae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ad69b289b4836937a7cd6f8182fae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } }
I20260927 11:13:21.364614 6006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:21.364540 6008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f7ad69b289b4836937a7cd6f8182fae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ad69b289b4836937a7cd6f8182fae" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36211 } } }
I20260927 11:13:21.364698 6008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:21.365093 6013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:21.365549 6013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:21.367291 6013 catalog_manager.cc:1347] Generated new cluster ID: 12512396ccb244058b331a0ba844691e
I20260927 11:13:21.367341 6013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:21.374045 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:21.364872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.0.98.62:36211)
0927 11:13:21.365032 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:21.365041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:21.365071 (+ 30us) server_negotiation.cc:407] Connection header received
0927 11:13:21.365186 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:21.365194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:21.365258 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:21.366242 (+ 984us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:21.366326 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.367495 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.368218 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.368627 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.368737 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:21.370972 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:21.370976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:21.370976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:21.370987 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:21.372595 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:21.372968 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:21.372971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:21.372971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:21.373061 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:21.373265 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:21.373266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:21.373267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:21.373484 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:21.373664 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:21.373727 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 11:13:21.373857 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260927 11:13:21.377512 6013 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:21.378037 6013 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:21.380932 6013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f7ad69b289b4836937a7cd6f8182fae: Generated new TSK 0
I20260927 11:13:21.381071 6013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:13:21.409253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36211
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:21.503759 6029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:21.503945 6029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:21.503968 6029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:21.505133 6029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:21.505172 6029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:21.505198 6029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:21.506721 6029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:21.506942 6029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:21.507167 6029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:21.511915 6034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:21.512068 6035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:21.511916 6037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:21.512216 6029 server_base.cc:1034] running on GCE node
I20260927 11:13:21.512475 6029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:21.512756 6029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:21.513903 6029 hybrid_clock.cc:648] HybridClock initialized: now 1790507601513889 us; error 40 us; skew 500 ppm
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:21.516391 6029 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:13:21.516777 6029 webserver.cc:460] Webserver started at http://127.0.98.1:37793/ using document root <none> and password file <none>
I20260927 11:13:21.516985 6029 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:21.517050 6029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:21.517194 6029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:21.518253 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "74353fb20c204ae5a66b96e660f54bc1"
format_stamp: "Formatted at 2026-09-27 11:13:21 on dist-test-slave-9hq9"
server_key: "03b9a3a6e9ed18a03bb2dabdae3bbeec"
server_key_iv: "15d12aa1fbb0e8395f4c65fc2f736dca"
server_key_version: "encryptionkey@0"
I20260927 11:13:21.518595 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "74353fb20c204ae5a66b96e660f54bc1"
format_stamp: "Formatted at 2026-09-27 11:13:21 on dist-test-slave-9hq9"
server_key: "03b9a3a6e9ed18a03bb2dabdae3bbeec"
server_key_iv: "15d12aa1fbb0e8395f4c65fc2f736dca"
server_key_version: "encryptionkey@0"
I20260927 11:13:21.519802 6029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:13:21.520459 6044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:21.520617 6029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:21.520711 6029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "74353fb20c204ae5a66b96e660f54bc1"
format_stamp: "Formatted at 2026-09-27 11:13:21 on dist-test-slave-9hq9"
server_key: "03b9a3a6e9ed18a03bb2dabdae3bbeec"
server_key_iv: "15d12aa1fbb0e8395f4c65fc2f736dca"
server_key_version: "encryptionkey@0"
I20260927 11:13:21.520784 6029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:21.542629 6029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:21.544354 6029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:21.544507 6029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:21.544720 6029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:21.545008 6029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:21.545059 6029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:21.545090 6029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:21.545125 6029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:21.550661 6029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36183
I20260927 11:13:21.550727 6157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36183 every 8 connection(s)
I20260927 11:13:21.551035 6029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:13:21.553922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 6029
I20260927 11:13:21.554014 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:13:21.554210 392 external_mini_cluster.cc:1428] Setting key 2993898cc3c7328a1198f097841194c6
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:21.561031 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:21.551733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45723 (local address 127.0.98.62:36211)
0927 11:13:21.551796 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:21.551798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:21.551970 (+ 172us) server_negotiation.cc:407] Connection header received
0927 11:13:21.552900 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:21.552902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:21.552918 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:21.552975 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:21.553538 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.554379 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.555501 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.555823 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.555894 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:21.558199 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:21.558200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:21.558200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:21.558208 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:21.559690 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:21.560094 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:21.560095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:21.560095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:21.560158 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:21.560415 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:21.560416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:21.560416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:21.560529 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:21.560604 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:21.560865 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 11:13:21.560945 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:13:21.561232 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:21.551716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36211 (local address 127.0.98.1:45723)
0927 11:13:21.551857 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:21.551897 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:21.552776 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:21.553036 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:21.553040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:21.553276 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:21.553443 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:21.553447 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.554500 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.554502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:21.555408 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:21.555411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.555912 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.555914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:21.556063 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:21.556495 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:21.556505 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:21.558013 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:21.559793 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:21.559799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:21.559801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:21.560016 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:21.560234 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:21.560235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:21.560235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:21.560341 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:21.560595 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:21.560601 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:21.560737 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 11:13:21.560843 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 11:13:21.560985 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 11:13:21.561697 6158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36211
I20260927 11:13:21.561782 6158 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:21.561944 6158 heartbeater.cc:511] Master 127.0.98.62:36211 requested a full tablet report, sending...
I20260927 11:13:21.562479 5967 ts_manager.cc:194] Registered new tserver with Master: 74353fb20c204ae5a66b96e660f54bc1 (127.0.98.1:36183)
I20260927 11:13:21.563124 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:13:21.577240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36211
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:21.655341 6163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:21.655510 6163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:21.655531 6163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:21.656536 6163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:21.656569 6163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:21.656592 6163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:21.657869 6163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:21.658020 6163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:21.658231 6163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:21.662717 6169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:21.662710 6168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:21.662711 6171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:21.663100 6163 server_base.cc:1034] running on GCE node
I20260927 11:13:21.663296 6163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:21.663473 6163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:21.664634 6163 hybrid_clock.cc:648] HybridClock initialized: now 1790507601664623 us; error 26 us; skew 500 ppm
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:21.667017 6163 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:13:21.667393 6163 webserver.cc:460] Webserver started at http://127.0.98.2:42445/ using document root <none> and password file <none>
I20260927 11:13:21.667589 6163 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:21.667629 6163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:21.667796 6163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:21.668761 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "92e633c6e5d4483ca0d1194ee34ff32f"
format_stamp: "Formatted at 2026-09-27 11:13:21 on dist-test-slave-9hq9"
server_key: "97eb4f82977b6cfe00e50385a1b583bb"
server_key_iv: "36f81add888144867554741a7add8d71"
server_key_version: "encryptionkey@0"
I20260927 11:13:21.669090 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "92e633c6e5d4483ca0d1194ee34ff32f"
format_stamp: "Formatted at 2026-09-27 11:13:21 on dist-test-slave-9hq9"
server_key: "97eb4f82977b6cfe00e50385a1b583bb"
server_key_iv: "36f81add888144867554741a7add8d71"
server_key_version: "encryptionkey@0"
I20260927 11:13:21.670370 6163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:13:21.671028 6178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:21.671200 6163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:21.671304 6163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "92e633c6e5d4483ca0d1194ee34ff32f"
format_stamp: "Formatted at 2026-09-27 11:13:21 on dist-test-slave-9hq9"
server_key: "97eb4f82977b6cfe00e50385a1b583bb"
server_key_iv: "36f81add888144867554741a7add8d71"
server_key_version: "encryptionkey@0"
I20260927 11:13:21.671382 6163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:21.686484 6163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:21.688148 6163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:21.688318 6163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:21.688527 6163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:21.688809 6163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:21.688860 6163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:21.688894 6163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:21.688921 6163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:21.694691 6163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32769
I20260927 11:13:21.694737 6291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32769 every 8 connection(s)
I20260927 11:13:21.695012 6163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:21.701648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 6163
I20260927 11:13:21.701764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:13:21.701924 392 external_mini_cluster.cc:1428] Setting key bdc165a8bd5146d42acf29af8b9fa991
I20260927 11:13:21.703467 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:21.695628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35253 (local address 127.0.98.62:36211)
0927 11:13:21.695670 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:21.695672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:21.695880 (+ 208us) server_negotiation.cc:407] Connection header received
0927 11:13:21.696696 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:21.696697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:21.696705 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:21.696758 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:21.697268 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.697725 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.698678 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.698854 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.698905 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:21.700694 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:21.700695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:21.700696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:21.700702 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:21.702166 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:21.702548 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:21.702548 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:21.702549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:21.702606 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:21.702838 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:21.702839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:21.702839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:21.702944 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:21.703041 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:21.703298 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 11:13:21.703377 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:13:21.703639 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:21.695596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36211 (local address 127.0.98.2:35253)
0927 11:13:21.695756 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:21.695794 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:21.696617 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:21.696823 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:21.696826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:21.697040 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:21.697202 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:21.697205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.697795 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.697795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:21.698610 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:21.698612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.698919 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.698919 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:21.699005 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:21.699382 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:21.699389 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:21.700611 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:21.702236 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:21.702242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:21.702244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:21.702485 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:21.702667 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:21.702668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:21.702668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:21.702766 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:21.703047 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:21.703053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:21.703177 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 11:13:21.703278 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 11:13:21.703430 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260927 11:13:21.704006 6292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36211
I20260927 11:13:21.704092 6292 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:21.704257 6292 heartbeater.cc:511] Master 127.0.98.62:36211 requested a full tablet report, sending...
I20260927 11:13:21.704629 5967 ts_manager.cc:194] Registered new tserver with Master: 92e633c6e5d4483ca0d1194ee34ff32f (127.0.98.2:32769)
I20260927 11:13:21.705140 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35253
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:13:21.728132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36211
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:21.806073 6297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:21.806296 6297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:21.806334 6297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:21.807461 6297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:21.807524 6297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:21.807569 6297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:21.809000 6297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:21.809248 6297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:21.809469 6297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:21.814113 6302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:21.814109 6303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:21.814217 6305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:21.814284 6297 server_base.cc:1034] running on GCE node
I20260927 11:13:21.814548 6297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:21.814777 6297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:21.815935 6297 hybrid_clock.cc:648] HybridClock initialized: now 1790507601815920 us; error 43 us; skew 500 ppm
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:21.818528 6297 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:13:21.818931 6297 webserver.cc:460] Webserver started at http://127.0.98.3:38991/ using document root <none> and password file <none>
I20260927 11:13:21.819092 6297 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:21.819128 6297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:21.819208 6297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:21.820113 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "81186992cd8e4ea6b9e468ecf70f000b"
format_stamp: "Formatted at 2026-09-27 11:13:21 on dist-test-slave-9hq9"
server_key: "cdaee825eabc1f3ba54fbbf661ed7d98"
server_key_iv: "b29727a8d75fe1729a22c3a68ef8a96e"
server_key_version: "encryptionkey@0"
I20260927 11:13:21.820415 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "81186992cd8e4ea6b9e468ecf70f000b"
format_stamp: "Formatted at 2026-09-27 11:13:21 on dist-test-slave-9hq9"
server_key: "cdaee825eabc1f3ba54fbbf661ed7d98"
server_key_iv: "b29727a8d75fe1729a22c3a68ef8a96e"
server_key_version: "encryptionkey@0"
I20260927 11:13:21.821569 6297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:13:21.822247 6312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:21.822399 6297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:21.822485 6297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "81186992cd8e4ea6b9e468ecf70f000b"
format_stamp: "Formatted at 2026-09-27 11:13:21 on dist-test-slave-9hq9"
server_key: "cdaee825eabc1f3ba54fbbf661ed7d98"
server_key_iv: "b29727a8d75fe1729a22c3a68ef8a96e"
server_key_version: "encryptionkey@0"
I20260927 11:13:21.822554 6297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:21.839604 6297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:21.841310 6297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:21.841457 6297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:21.841660 6297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:21.841950 6297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:21.841981 6297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:21.842003 6297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:21.842079 6297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:21.847656 6297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33389
I20260927 11:13:21.847695 6425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33389 every 8 connection(s)
I20260927 11:13:21.847980 6297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:13:21.852468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 6297
I20260927 11:13:21.852572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:13:21.852763 392 external_mini_cluster.cc:1428] Setting key e784c20fc09635118f6591dc4bc757b2
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:21.856348 6017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:21.848591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46615 (local address 127.0.98.62:36211)
0927 11:13:21.848629 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:21.848631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:21.848839 (+ 208us) server_negotiation.cc:407] Connection header received
0927 11:13:21.849693 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:21.849694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:21.849703 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:21.849760 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:21.850359 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.850843 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.851751 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.851921 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.851968 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:21.854237 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:21.854239 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:21.854239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:21.854245 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:21.855346 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:21.855685 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:21.855685 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:21.855686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:21.855740 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:21.855912 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:21.855912 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:21.855912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:21.855997 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:21.856044 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:21.856238 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 11:13:21.856290 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:13:21.856508 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:21.848586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36211 (local address 127.0.98.3:46615)
0927 11:13:21.848722 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:21.848753 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:21.849610 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:21.849812 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:21.849815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:21.850045 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:21.850292 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:21.850296 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.850912 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.850913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:21.851683 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:21.851685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.851985 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.851986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:21.852074 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:21.852481 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:21.852491 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:21.854160 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:21.855409 (+ 1249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:21.855416 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:21.855418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:21.855623 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:21.855785 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:21.855785 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:21.855785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:21.855873 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:21.856038 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:21.856040 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:21.856132 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 11:13:21.856227 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 11:13:21.856334 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":87,"threads_started":1}
I20260927 11:13:21.856957 6426 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36211
I20260927 11:13:21.857029 6426 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:21.857214 6426 heartbeater.cc:511] Master 127.0.98.62:36211 requested a full tablet report, sending...
I20260927 11:13:21.857573 5967 ts_manager.cc:194] Registered new tserver with Master: 81186992cd8e4ea6b9e468ecf70f000b (127.0.98.3:33389)
I20260927 11:13:21.857939 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46615
I20260927 11:13:21.864389 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/test-admin.keytab.
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:21.878379 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 11:13:21.879850 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:21.879147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.0.98.1:36183)
0927 11:13:21.879284 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:21.879295 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:21.879305 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:13:21.879327 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:21.879329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:21.879343 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:21.879404 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:21.879744 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52908: BlockingRecv error: recv got EOF from 127.0.0.1:52908 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
W20260927 11:13:21.879995 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:21.880957 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:21.881901 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:13:21.889276 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:21.882418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52914 (local address 127.0.98.1:36183)
0927 11:13:21.882485 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:21.882487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:21.882497 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:13:21.882560 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:21.882561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:21.882570 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:21.882637 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:21.883018 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.883709 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.884426 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.884733 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.884819 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:21.886806 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:21.886814 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:21.886814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:21.886821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:21.888122 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:21.888473 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:21.888475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:21.888476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:21.888535 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:21.888732 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:21.888733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:21.888733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:21.888855 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:21.889020 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:21.889099 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 11:13:21.889168 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 11:13:21.890928 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:21.890431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.0.98.1:36183)
0927 11:13:21.890484 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:21.890486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:21.890509 (+ 23us) server_negotiation.cc:407] Connection header received
0927 11:13:21.890577 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:21.890578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:21.890587 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:21.890634 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:21.890885 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52920: BlockingRecv error: recv got EOF from 127.0.0.1:52920 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 11:13:21.891016 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:21.891870 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:21.892643 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:13:21.898622 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:21.893095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.0.98.1:36183)
0927 11:13:21.893148 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:21.893150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:21.893206 (+ 56us) server_negotiation.cc:407] Connection header received
0927 11:13:21.893229 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:21.893229 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:21.893237 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:21.893275 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:21.893584 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.894030 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.894696 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.894883 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.894931 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:21.896608 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:21.896609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:21.896610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:21.896615 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:21.897690 (+ 1075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:21.897964 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:21.897965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:21.897965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:21.898009 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:21.898215 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:21.898215 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:21.898215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:21.898292 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:21.898355 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:21.898446 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 11:13:21.898496 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 11:13:21.899876 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:21.899433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52928 (local address 127.0.98.1:36183)
0927 11:13:21.899465 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:21.899471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:21.899511 (+ 40us) server_negotiation.cc:407] Connection header received
0927 11:13:21.899564 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:21.899565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:21.899571 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:21.899615 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:21.899833 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52928: BlockingRecv error: recv got EOF from 127.0.0.1:52928 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 11:13:21.899991 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790507576023819-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:21.900875 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:21.901662 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 11:13:21 dist-test-slave-9hq9 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:13:21.907740 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:21.902121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52932 (local address 127.0.98.1:36183)
0927 11:13:21.902174 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:21.902175 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:21.902201 (+ 26us) server_negotiation.cc:407] Connection header received
0927 11:13:21.902253 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:21.902254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:21.902260 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:21.902301 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:21.902580 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.903011 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.903614 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:21.903779 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:21.903824 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:21.905530 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:21.905531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:21.905531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:21.905537 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:21.906913 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:21.907177 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:21.907177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:21.907177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:21.907227 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:21.907362 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:21.907362 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:21.907362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:21.907432 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:21.907500 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:21.907570 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 11:13:21.907617 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:13:21.908229 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 6029
I20260927 11:13:21.913086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 6163
I20260927 11:13:21.916910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 6297
I20260927 11:13:21.920933 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 5935
2026-09-27T11:13:21Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1808 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:13:22 dist-test-slave-9hq9 krb5kdc[6446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:13:22 dist-test-slave-9hq9 krb5kdc[6446](info): set up 2 sockets
Sep 27 11:13:22 dist-test-slave-9hq9 krb5kdc[6446](info): commencing operation
krb5kdc: starting...
W20260927 11:13:23.290225 392 mini_kdc.cc:121] Time spent starting KDC: real 1.349s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:13:23 dist-test-slave-9hq9 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:13:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 11:13:23.397013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45085
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:23.472661 6459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:23.472849 6459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:23.472872 6459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:23.473986 6459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:23.474026 6459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:23.474040 6459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:23.474099 6459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:23.474140 6459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:23.475562 6459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:23.475788 6459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:23.476024 6459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:13:23.480796 6459 server_base.cc:1034] running on GCE node
W20260927 11:13:23.480857 6467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:23.480974 6464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:23.480800 6465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:23.481283 6459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:23.481563 6459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:23.482712 6459 hybrid_clock.cc:648] HybridClock initialized: now 1790507603482690 us; error 35 us; skew 500 ppm
Sep 27 11:13:23 dist-test-slave-9hq9 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507603, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:23.484931 6459 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 11:13:23.485351 6459 webserver.cc:460] Webserver started at http://127.0.98.62:34279/ using document root <none> and password file <none>
I20260927 11:13:23.485577 6459 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:23.485649 6459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:23.485795 6459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:23.486811 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "e3930554fc654a9eb440e352ed32c2df"
format_stamp: "Formatted at 2026-09-27 11:13:23 on dist-test-slave-9hq9"
server_key: "36645349ff6bf0d4e8c265cb683b4fb5"
server_key_iv: "83e6705f26f4489616f31d479dfb199f"
server_key_version: "encryptionkey@0"
I20260927 11:13:23.487150 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "e3930554fc654a9eb440e352ed32c2df"
format_stamp: "Formatted at 2026-09-27 11:13:23 on dist-test-slave-9hq9"
server_key: "36645349ff6bf0d4e8c265cb683b4fb5"
server_key_iv: "83e6705f26f4489616f31d479dfb199f"
server_key_version: "encryptionkey@0"
I20260927 11:13:23.488353 6459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:13:23.489006 6474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:23.489169 6459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:23.489269 6459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "e3930554fc654a9eb440e352ed32c2df"
format_stamp: "Formatted at 2026-09-27 11:13:23 on dist-test-slave-9hq9"
server_key: "36645349ff6bf0d4e8c265cb683b4fb5"
server_key_iv: "83e6705f26f4489616f31d479dfb199f"
server_key_version: "encryptionkey@0"
I20260927 11:13:23.489339 6459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:23.499197 6459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:23.500805 6459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:23.500967 6459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:23.504829 6459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45085
I20260927 11:13:23.504855 6526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45085 every 8 connection(s)
I20260927 11:13:23.505134 6459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:23.505976 6527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:23.507967 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df: Bootstrap starting.
I20260927 11:13:23.508441 6527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:23.508656 6527 log.cc:824] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:23.509239 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df: No bootstrap required, opened a new log
I20260927 11:13:23.510696 6527 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } }
I20260927 11:13:23.510866 6527 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:23.510910 6527 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3930554fc654a9eb440e352ed32c2df, State: Initialized, Role: FOLLOWER
I20260927 11:13:23.511023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 6459
I20260927 11:13:23.511056 6527 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } }
I20260927 11:13:23.511135 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:23.511162 6527 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:23.511196 6527 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:23.511247 6527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:23.511315 392 external_mini_cluster.cc:1428] Setting key 1c4e7963d541dafec2e84fe14211659f
I20260927 11:13:23.512245 6527 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } }
I20260927 11:13:23.512360 6527 leader_election.cc:302] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3930554fc654a9eb440e352ed32c2df; no voters:
I20260927 11:13:23.512509 6527 leader_election.cc:288] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:23.512566 6532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:23.512723 6532 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [term 1 LEADER]: Becoming Leader. State: Replica: e3930554fc654a9eb440e352ed32c2df, State: Running, Role: LEADER
I20260927 11:13:23.512745 6527 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:23.512807 6532 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } }
I20260927 11:13:23.513155 6532 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3930554fc654a9eb440e352ed32c2df. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } }
I20260927 11:13:23.513283 6532 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:23.513824 6540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:23.514398 6540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:23.514846 6533 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } }
I20260927 11:13:23.514981 6533 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:23.515841 6540 catalog_manager.cc:1347] Generated new cluster ID: b56755796adc430a96201df52bbd5158
I20260927 11:13:23.515888 6540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:13:23 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:13:23.521042 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:23.511757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40818 (local address 127.0.98.62:45085)
0927 11:13:23.511929 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:23.511937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:23.511972 (+ 35us) server_negotiation.cc:407] Connection header received
0927 11:13:23.512052 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:23.512057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:23.512093 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:23.512206 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:23.513041 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:23.514526 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:23.515529 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:23.516069 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:23.516199 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:23.518317 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:23.518320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:23.518320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:23.518331 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:23.519574 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:23.519908 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:23.519910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:23.519911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:23.520006 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:23.520221 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:23.520222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:23.520223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:23.520434 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:23.520600 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:23.520679 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 11:13:23.520806 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260927 11:13:23.527676 6540 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:23.528127 6540 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:23.534577 6540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df: Generated new TSK 0
I20260927 11:13:23.534761 6540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 11:13:23.558306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45085
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:23.634673 6553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:23.634884 6553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:23.634920 6553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:23.635985 6553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:23.636049 6553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:23.636092 6553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:23.637488 6553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:23.637742 6553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:23.637951 6553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:23.642831 6561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:23.642884 6558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:23.642942 6559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:23.642886 6553 server_base.cc:1034] running on GCE node
I20260927 11:13:23.643234 6553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:23.643457 6553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:23.644616 6553 hybrid_clock.cc:648] HybridClock initialized: now 1790507603644602 us; error 27 us; skew 500 ppm
Sep 27 11:13:23 dist-test-slave-9hq9 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507603, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:23.646926 6553 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 11:13:23.647343 6553 webserver.cc:460] Webserver started at http://127.0.98.1:40783/ using document root <none> and password file <none>
I20260927 11:13:23.647516 6553 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:23.647553 6553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:23.647660 6553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:23.648537 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "ff5b9fa623da4e6cb3389c7f86669d63"
format_stamp: "Formatted at 2026-09-27 11:13:23 on dist-test-slave-9hq9"
server_key: "690e508531e3a08c67bb75b2bd5155c2"
server_key_iv: "19eee59ec649247f7e6b365adb6c9a58"
server_key_version: "encryptionkey@0"
I20260927 11:13:23.648828 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ff5b9fa623da4e6cb3389c7f86669d63"
format_stamp: "Formatted at 2026-09-27 11:13:23 on dist-test-slave-9hq9"
server_key: "690e508531e3a08c67bb75b2bd5155c2"
server_key_iv: "19eee59ec649247f7e6b365adb6c9a58"
server_key_version: "encryptionkey@0"
I20260927 11:13:23.649978 6553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:13:23.650657 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:23.650803 6553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:23.650882 6553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "ff5b9fa623da4e6cb3389c7f86669d63"
format_stamp: "Formatted at 2026-09-27 11:13:23 on dist-test-slave-9hq9"
server_key: "690e508531e3a08c67bb75b2bd5155c2"
server_key_iv: "19eee59ec649247f7e6b365adb6c9a58"
server_key_version: "encryptionkey@0"
I20260927 11:13:23.650933 6553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:23.672657 6553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:23.674803 6553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:23.674901 6553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:23.675058 6553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:23.675264 6553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:23.675292 6553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:23.675310 6553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:23.675323 6553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:23.680685 6553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36085
I20260927 11:13:23.680783 6681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36085 every 8 connection(s)
I20260927 11:13:23.681030 6553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:13:23.682250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 6553
I20260927 11:13:23.682341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:13:23.682536 392 external_mini_cluster.cc:1428] Setting key 43247aaf1bc98aa64d915f98977b7fe8
Sep 27 11:13:23 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507603, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 11:13:23.690904 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:23.681654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33481 (local address 127.0.98.62:45085)
0927 11:13:23.681693 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:23.681695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:23.681902 (+ 207us) server_negotiation.cc:407] Connection header received
0927 11:13:23.682625 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:23.682627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:23.682642 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:23.682714 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:23.683941 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:23.684502 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:23.685533 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:23.685904 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:23.685976 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:23.687959 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:23.687960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:23.687961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:23.687968 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:23.689595 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:23.689941 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:23.689942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:23.689943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:23.690001 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:23.690284 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:23.690285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:23.690285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:23.690423 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:23.690495 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:23.690704 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 11:13:23.690786 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:13:23.690984 6684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:23.681621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.1:33481)
0927 11:13:23.681756 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:23.681819 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:23.682479 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:23.682814 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:23.682816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:23.683584 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:23.683852 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:23.683858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:23.684609 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:23.684611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:23.685437 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:23.685439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:23.685980 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:23.685981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:23.686123 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:23.686637 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:23.686646 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:23.687875 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:23.689680 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:23.689685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:23.689687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:23.689872 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:23.690122 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:23.690122 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:23.690123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:23.690213 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:23.690490 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:23.690494 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:23.690586 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 11:13:23.690660 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 11:13:23.690760 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260927 11:13:23.691461 6682 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45085
I20260927 11:13:23.691538 6682 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:23.691715 6682 heartbeater.cc:511] Master 127.0.98.62:45085 requested a full tablet report, sending...
I20260927 11:13:23.692185 6491 ts_manager.cc:194] Registered new tserver with Master: ff5b9fa623da4e6cb3389c7f86669d63 (127.0.98.1:36085)
I20260927 11:13:23.692699 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 11:13:23.706885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45085
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:23.790997 6687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:23.791188 6687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:23.791209 6687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:23.793078 6687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:23.793131 6687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:23.793179 6687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:23.795045 6687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:23.795295 6687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:23.795511 6687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:23.800446 6693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:23.800498 6695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:23.800474 6692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:23.800647 6687 server_base.cc:1034] running on GCE node
I20260927 11:13:23.800838 6687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:23.801067 6687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:23.802217 6687 hybrid_clock.cc:648] HybridClock initialized: now 1790507603802191 us; error 39 us; skew 500 ppm
Sep 27 11:13:23 dist-test-slave-9hq9 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507603, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:23.804531 6687 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 11:13:23.804872 6687 webserver.cc:460] Webserver started at http://127.0.98.2:45829/ using document root <none> and password file <none>
I20260927 11:13:23.805032 6687 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:23.805069 6687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:23.805158 6687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:23.806114 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "528b23b21f864e4dacae07daace39679"
format_stamp: "Formatted at 2026-09-27 11:13:23 on dist-test-slave-9hq9"
server_key: "d964891fc48dd99319d9aade7c06be43"
server_key_iv: "c627edce7656f80c6f38bd03b549aee1"
server_key_version: "encryptionkey@0"
I20260927 11:13:23.806419 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "528b23b21f864e4dacae07daace39679"
format_stamp: "Formatted at 2026-09-27 11:13:23 on dist-test-slave-9hq9"
server_key: "d964891fc48dd99319d9aade7c06be43"
server_key_iv: "c627edce7656f80c6f38bd03b549aee1"
server_key_version: "encryptionkey@0"
I20260927 11:13:23.807682 6687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:23.808280 6702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:23.808436 6687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:23.808513 6687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "528b23b21f864e4dacae07daace39679"
format_stamp: "Formatted at 2026-09-27 11:13:23 on dist-test-slave-9hq9"
server_key: "d964891fc48dd99319d9aade7c06be43"
server_key_iv: "c627edce7656f80c6f38bd03b549aee1"
server_key_version: "encryptionkey@0"
I20260927 11:13:23.808588 6687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:23.847378 6687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:23.849077 6687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:23.849231 6687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:23.849442 6687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:23.849730 6687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:23.849761 6687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:23.849781 6687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:23.849828 6687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:23.855427 6687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34545
I20260927 11:13:23.855470 6815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34545 every 8 connection(s)
I20260927 11:13:23.855731 6687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:13:23 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507603, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:13:23.863396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 6687
I20260927 11:13:23.863507 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:13:23.863735 392 external_mini_cluster.cc:1428] Setting key f34ea335eea7f3b933f380f4562c9469
I20260927 11:13:23.864792 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:23.856353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51621 (local address 127.0.98.62:45085)
0927 11:13:23.856403 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:23.856405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:23.856563 (+ 158us) server_negotiation.cc:407] Connection header received
0927 11:13:23.856951 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:23.856952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:23.856961 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:23.857002 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:23.857818 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:23.858405 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:23.859436 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:23.859612 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:23.859662 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:23.861492 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:23.861493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:23.861493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:23.861499 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:23.863184 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:23.863615 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:23.863616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:23.863617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:23.863673 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:23.863966 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:23.863967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:23.863967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:23.864126 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:23.864203 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:23.864507 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 11:13:23.864629 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:13:23.865061 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:23.856329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.2:51621)
0927 11:13:23.856461 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:23.856490 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:23.856878 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:23.857073 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:23.857076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:23.857585 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:23.857747 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:23.857751 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:23.858505 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:23.858506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:23.859369 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:23.859372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:23.859676 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:23.859676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:23.859762 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:23.860162 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:23.860169 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:23.861415 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:23.863256 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:23.863262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:23.863264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:23.863510 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:23.863742 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:23.863742 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:23.863743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:23.863862 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:23.864201 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:23.864206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:23.864344 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 11:13:23.864443 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 11:13:23.864597 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":80,"threads_started":1}
I20260927 11:13:23.866189 6816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45085
I20260927 11:13:23.866288 6816 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:23.866460 6816 heartbeater.cc:511] Master 127.0.98.62:45085 requested a full tablet report, sending...
I20260927 11:13:23.866876 6491 ts_manager.cc:194] Registered new tserver with Master: 528b23b21f864e4dacae07daace39679 (127.0.98.2:34545)
I20260927 11:13:23.867249 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51621
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 11:13:23.894809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45085
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:23.973096 6821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:23.973276 6821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:23.973299 6821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:23.974337 6821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:23.974370 6821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:23.974395 6821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:23.975673 6821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:23.975827 6821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:23.975971 6821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:23.980830 6826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:23.980924 6821 server_base.cc:1034] running on GCE node
W20260927 11:13:23.980932 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:23.980952 6829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:23.981277 6821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:23.981514 6821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:23.982676 6821 hybrid_clock.cc:648] HybridClock initialized: now 1790507603982667 us; error 30 us; skew 500 ppm
Sep 27 11:13:23 dist-test-slave-9hq9 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507603, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:23.984941 6821 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 11:13:23.985285 6821 webserver.cc:460] Webserver started at http://127.0.98.3:43771/ using document root <none> and password file <none>
I20260927 11:13:23.985460 6821 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:23.985497 6821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:23.985576 6821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:23.986512 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "24208615a2e54f9d9638016054d89e17"
format_stamp: "Formatted at 2026-09-27 11:13:23 on dist-test-slave-9hq9"
server_key: "644134a097358c70cf2dcb1a781852f0"
server_key_iv: "314747e47df016808fc67e022488568c"
server_key_version: "encryptionkey@0"
I20260927 11:13:23.986794 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "24208615a2e54f9d9638016054d89e17"
format_stamp: "Formatted at 2026-09-27 11:13:23 on dist-test-slave-9hq9"
server_key: "644134a097358c70cf2dcb1a781852f0"
server_key_iv: "314747e47df016808fc67e022488568c"
server_key_version: "encryptionkey@0"
I20260927 11:13:23.987998 6821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:13:23.988641 6836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:23.988799 6821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:23.988890 6821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "24208615a2e54f9d9638016054d89e17"
format_stamp: "Formatted at 2026-09-27 11:13:23 on dist-test-slave-9hq9"
server_key: "644134a097358c70cf2dcb1a781852f0"
server_key_iv: "314747e47df016808fc67e022488568c"
server_key_version: "encryptionkey@0"
I20260927 11:13:23.988966 6821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:24.008493 6821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:24.010190 6821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:24.010336 6821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:24.010526 6821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:24.010826 6821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:24.010931 6821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.010986 6821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:24.011016 6821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.016979 6821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38171
I20260927 11:13:24.017017 6949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38171 every 8 connection(s)
I20260927 11:13:24.017290 6821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:13:24.018798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 6821
I20260927 11:13:24.018903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:13:24.019073 392 external_mini_cluster.cc:1428] Setting key 4e6b1e8abd1fa65ae507e130523278da
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507603, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:13:24.025344 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.017865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48691 (local address 127.0.98.62:45085)
0927 11:13:24.017924 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.017926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.018131 (+ 205us) server_negotiation.cc:407] Connection header received
0927 11:13:24.018579 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.018580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.018592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.018632 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.019696 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.020207 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.021094 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.021281 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.021332 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.023140 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.023142 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.023142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.023148 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.024283 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.024604 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.024605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.024605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.024648 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.024842 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.024843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.024843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.024930 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.024975 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.025213 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.025286 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 11:13:24.025511 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.017838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.3:48691)
0927 11:13:24.017994 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.018034 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.018484 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.018703 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.018707 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.019370 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.019618 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.019624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.020279 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.020280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.021023 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.021024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.021348 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.021349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.021432 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.021780 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.021787 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.023066 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.024347 (+ 1281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.024352 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.024353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.024536 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.024701 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.024702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.024702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.024785 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.024979 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.024982 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.025115 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.025187 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.025303 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260927 11:13:24.025993 6950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45085
I20260927 11:13:24.026126 6950 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:24.026288 6950 heartbeater.cc:511] Master 127.0.98.62:45085 requested a full tablet report, sending...
I20260927 11:13:24.026638 6491 ts_manager.cc:194] Registered new tserver with Master: 24208615a2e54f9d9638016054d89e17 (127.0.98.3:38171)
I20260927 11:13:24.026975 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48691
I20260927 11:13:24.030560 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 11:13:24.053213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40661
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45085,127.0.98.61:40661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:24.130162 6955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:24.130339 6955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:24.130360 6955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:24.131485 6955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:24.131525 6955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:24.131539 6955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:24.131552 6955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:24.131603 6955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:24.132980 6955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45085,127.0.98.61:40661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
W20260927 11:13:24.133102 6955 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:45085,127.0.98.61:40661'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 11:13:24.133239 6955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:24.133471 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:24.137961 6961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:24.138015 6963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:24.138132 6955 server_base.cc:1034] running on GCE node
W20260927 11:13:24.137953 6960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:24.138404 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:24.138697 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:24.139837 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790507604139835 us; error 34 us; skew 500 ppm
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:24.143144 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 11:13:24.143584 6955 webserver.cc:460] Webserver started at http://127.0.98.61:39333/ using document root <none> and password file <none>
I20260927 11:13:24.143842 6955 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:24.143887 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:24.144038 6955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:24.145226 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/data/instance:
uuid: "bfc088fef7a4465e8a6176585e798392"
format_stamp: "Formatted at 2026-09-27 11:13:24 on dist-test-slave-9hq9"
server_key: "927ceba1c083985bd879ad3f7fc2fdd9"
server_key_iv: "1bfb58f558064263cf14fcfb0bd723cb"
server_key_version: "encryptionkey@0"
I20260927 11:13:24.145666 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/wal/instance:
uuid: "bfc088fef7a4465e8a6176585e798392"
format_stamp: "Formatted at 2026-09-27 11:13:24 on dist-test-slave-9hq9"
server_key: "927ceba1c083985bd879ad3f7fc2fdd9"
server_key_iv: "1bfb58f558064263cf14fcfb0bd723cb"
server_key_version: "encryptionkey@0"
I20260927 11:13:24.147457 6955 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 11:13:24.148430 6970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.148603 6955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:24.148693 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/wal
uuid: "bfc088fef7a4465e8a6176585e798392"
format_stamp: "Formatted at 2026-09-27 11:13:24 on dist-test-slave-9hq9"
server_key: "927ceba1c083985bd879ad3f7fc2fdd9"
server_key_iv: "1bfb58f558064263cf14fcfb0bd723cb"
server_key_version: "encryptionkey@0"
I20260927 11:13:24.148770 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:24.176503 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:24.178251 6955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:24.178351 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:24.181808 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40661
I20260927 11:13:24.181850 7022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40661 every 8 connection(s)
I20260927 11:13:24.182235 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/data/info.pb
I20260927 11:13:24.183161 7023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:24.184473 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:13:24.187170 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 6955
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 11:13:24.193307 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.185064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40832 (local address 127.0.98.62:45085)
0927 11:13:24.185128 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.185130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.185291 (+ 161us) server_negotiation.cc:407] Connection header received
0927 11:13:24.185719 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.185719 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.185728 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.185781 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.186345 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.186812 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.187953 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.188194 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.188244 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.190307 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.190308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.190308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.190315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.191949 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.192400 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.192401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.192402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.192459 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.192680 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.192681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.192682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.192861 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.192936 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.193192 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.193248 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 11:13:24.193449 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.185035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:40832)
0927 11:13:24.185178 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.185214 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.185641 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.185828 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.185830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.186035 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.186279 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.186283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.186883 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.186883 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.187879 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.187882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.188273 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.188274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.188443 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.188929 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.188937 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.190236 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.192016 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.192021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.192023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.192314 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.192520 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.192521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.192521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.192616 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.192938 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.192944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.193080 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.193160 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.193258 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260927 11:13:24.194015 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:13:24.196815 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.187633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52106 (local address 127.0.98.61:40661)
0927 11:13:24.187796 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.187800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.187821 (+ 21us) server_negotiation.cc:407] Connection header received
0927 11:13:24.187880 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.187883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.187905 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.187983 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.188652 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.189400 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.190250 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.190686 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.190755 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.193930 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.193936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.193936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.193942 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.195757 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.196114 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.196115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.196116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.196162 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.196344 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.196344 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.196344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.196450 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.196539 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.196657 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.196723 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 11:13:24.201172 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.194272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40661 (local address 127.0.0.1:52110)
0927 11:13:24.194315 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.194324 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.194362 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.194642 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.194643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.194853 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.194898 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.194900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.195817 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.195819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.196565 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.196567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.197036 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.197037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.197105 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.197463 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.197467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.198532 (+ 1065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.200481 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.200482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.200482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.200619 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.200774 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.200774 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.200775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.200818 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.200974 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.200975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.201003 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.201056 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.201120 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 11:13:24.201282 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.194366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.0.98.61:40661)
0927 11:13:24.194536 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.194537 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.194543 (+ 6us) server_negotiation.cc:407] Connection header received
0927 11:13:24.194556 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.194557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.194564 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.194641 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.194980 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.195714 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.196636 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.196958 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.197033 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.198604 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.198605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.198605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.198610 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.200419 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.200676 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.200676 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.200677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.200722 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.200854 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.200855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.200855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.200930 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.200971 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.201069 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.201163 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":147,"thread_start_us":65,"threads_started":1}
I20260927 11:13:24.202467 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: Bootstrap starting.
I20260927 11:13:24.203014 7023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:24.203279 7023 log.cc:824] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:24.203866 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: No bootstrap required, opened a new log
I20260927 11:13:24.205090 7023 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } }
I20260927 11:13:24.205214 7023 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:24.205259 7023 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfc088fef7a4465e8a6176585e798392, State: Initialized, Role: FOLLOWER
I20260927 11:13:24.205399 7023 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } }
I20260927 11:13:24.205655 7023 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:24.205636 7030 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } } }
I20260927 11:13:24.205770 7030 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 11:13:24.207150 7041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:13:24.207206 7041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:13:24.208604 6955 master_runner.cc:428] Detected that this master bfc088fef7a4465e8a6176585e798392 is joining an existing cluster
I20260927 11:13:24.208657 6955 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:40661
I20260927 11:13:24.217042 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.209546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40661 (local address 127.0.0.1:52114)
0927 11:13:24.209673 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.209682 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.209741 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.209856 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.209857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.210089 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.210157 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.210159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.211049 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.211050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.211933 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.211935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.212481 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.212482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.212543 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.212906 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.212910 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.213250 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.215851 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.215852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.215852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.216109 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.216331 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.216332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.216332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.216396 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.216700 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.216701 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.216759 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.216836 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.216931 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260927 11:13:24.217278 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.209454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.0.98.61:40661)
0927 11:13:24.209488 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.209490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.209747 (+ 257us) server_negotiation.cc:407] Connection header received
0927 11:13:24.209790 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.209791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.209800 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.209853 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.210232 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.210962 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.212011 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.212282 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.212326 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.213316 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.213317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.213318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.213324 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.215762 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.216198 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.216199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.216200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.216256 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.216473 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.216474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.216474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.216626 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.216688 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.216864 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.216980 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":17}
I20260927 11:13:24.217517 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.209481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:40840)
0927 11:13:24.209631 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.209642 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.209718 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.209910 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.209910 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.210144 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.210189 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.210190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.211115 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.211116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.211734 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.211735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.212157 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.212158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.212215 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.213349 (+ 1134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.213353 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.213744 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.216467 (+ 2723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.216468 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.216468 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.216637 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.216884 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.216885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.216885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.216943 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.217214 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.217215 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.217250 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.217327 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.217415 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260927 11:13:24.217564 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.209468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.0.98.62:45085)
0927 11:13:24.209506 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.209508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.209699 (+ 191us) server_negotiation.cc:407] Connection header received
0927 11:13:24.209808 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.209809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.209818 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.209848 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.210296 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.211009 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.211810 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.212052 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.212138 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.213819 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.213820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.213821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.213827 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.216372 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.216748 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.216749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.216750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.216809 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.217012 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.217012 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.217013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.217125 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.217166 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.217388 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.217481 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507603, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 11:13:24.226083 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.219454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.0.98.61:40661)
0927 11:13:24.219488 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.219490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.219615 (+ 125us) server_negotiation.cc:407] Connection header received
0927 11:13:24.219696 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.219697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.219704 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.219742 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.220164 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.220560 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.221561 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.221715 (+ 154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.221759 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.223698 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.223699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.223700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.223706 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.224895 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.225269 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.225270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.225270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.225319 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.225563 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.225564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.225564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.225658 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.225731 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.225914 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.225982 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:13:24.226085 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.219376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40661 (local address 127.0.0.1:52122)
0927 11:13:24.219513 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.219555 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.219630 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.219756 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.219758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.220052 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.220098 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.220099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.220631 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.220632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.221487 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.221489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.221780 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.221781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.221866 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.222277 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.222281 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.223612 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.224989 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.224995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.224997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.225185 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.225431 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.225432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.225432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.225499 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.225726 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.225732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.225781 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.225878 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.225984 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260927 11:13:24.226608 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:40661: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bfc088fef7a4465e8a6176585e798392"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 40661
}
attrs {
promote: true
}
}
dest_uuid: "e3930554fc654a9eb440e352ed32c2df"
cas_config_opid_index: -1
I20260927 11:13:24.226893 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: true } }
I20260927 11:13:24.227304 7051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER bfc088fef7a4465e8a6176585e798392 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: true } } }
I20260927 11:13:24.227615 6997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:24.228288 7051 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:40661
I20260927 11:13:24.228363 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: true } } }
I20260927 11:13:24.228408 6997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 FOLLOWER]: Refusing update from remote peer e3930554fc654a9eb440e352ed32c2df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 11:13:24.228441 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:24.228828 6955 master.cc:537] Master@127.0.98.61:40661 shutting down...
I20260927 11:13:24.228832 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 11:13:24.230114 6476 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 11:13:24.230527 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df -> Peer bfc088fef7a4465e8a6176585e798392 (127.0.98.61:40661): Couldn't send request to peer bfc088fef7a4465e8a6176585e798392. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 11:13:24.231417 6955 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:24.231480 6955 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:24.231536 6955 tablet_replica.cc:330] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: stopping tablet replica
I20260927 11:13:24.243547 6955 master.cc:552] Master@127.0.98.61:40661 shutdown complete.
I20260927 11:13:24.243583 6955 master_runner.cc:305] Clearing existing system tablet
I20260927 11:13:24.243716 6955 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:24.243752 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:24.244331 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:24.244845 7055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.244971 6955 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:24.245024 6955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.245046 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/wal
uuid: "bfc088fef7a4465e8a6176585e798392"
format_stamp: "Formatted at 2026-09-27 11:13:24 on dist-test-slave-9hq9"
server_key: "927ceba1c083985bd879ad3f7fc2fdd9"
server_key_iv: "1bfb58f558064263cf14fcfb0bd723cb"
server_key_version: "encryptionkey@0"
I20260927 11:13:24.245585 6955 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:24.246836 6955 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 11:13:24.246889 6955 log.cc:1196] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 11:13:24.247079 6955 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: Deleting consensus metadata
I20260927 11:13:24.247197 6955 master_runner.cc:315] Copying system tablet from 127.0.98.62:45085
I20260927 11:13:24.247778 6955 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45085
I20260927 11:13:24.253108 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.248094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.0.98.62:45085)
0927 11:13:24.248156 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.248158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.248187 (+ 29us) server_negotiation.cc:407] Connection header received
0927 11:13:24.248259 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.248260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.248269 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.248323 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.248604 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.249063 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.249750 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.249950 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.250019 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.250848 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.250849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.250850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.250856 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.252049 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.252380 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.252381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.252382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.252439 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.252640 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.252641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.252642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.252743 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.252815 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.252958 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.253025 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 11:13:24.253131 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.248031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:40844)
0927 11:13:24.248129 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.248140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.248203 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.248328 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.248329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.248486 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.248530 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.248531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.249150 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.249151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.249670 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.249671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.250030 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.250031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.250168 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.250535 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.250538 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.250765 (+ 227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.252128 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.252129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.252130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.252309 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.252511 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.252511 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.252512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.252568 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.252808 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.252809 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.252848 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.252935 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.253027 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":61,"threads_started":1}
I20260927 11:13:24.253460 6511 tablet_copy_service.cc:140] P e3930554fc654a9eb440e352ed32c2df: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bfc088fef7a4465e8a6176585e798392 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40844)
I20260927 11:13:24.253531 6511 tablet_copy_service.cc:161] P e3930554fc654a9eb440e352ed32c2df: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bfc088fef7a4465e8a6176585e798392 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40844: session id = bfc088fef7a4465e8a6176585e798392-00000000000000000000000000000000
I20260927 11:13:24.254127 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 11:13:24.254647 6955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 11:13:24.255360 6955 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: tablet copy: Starting download of 0 data blocks...
I20260927 11:13:24.255497 6955 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: tablet copy: Starting download of 1 WAL segments...
I20260927 11:13:24.256222 6955 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 11:13:24.256882 6511 tablet_copy_service.cc:342] P e3930554fc654a9eb440e352ed32c2df: Request end of tablet copy session bfc088fef7a4465e8a6176585e798392-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40844
I20260927 11:13:24.256942 6511 tablet_copy_service.cc:433] P e3930554fc654a9eb440e352ed32c2df: ending tablet copy session bfc088fef7a4465e8a6176585e798392-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bfc088fef7a4465e8a6176585e798392
I20260927 11:13:24.258939 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:24.259969 7062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:24.260056 7065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:24.260118 7063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:24.260291 6955 server_base.cc:1034] running on GCE node
I20260927 11:13:24.260367 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:24.260473 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:24.261598 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790507604261595 us; error 30 us; skew 500 ppm
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:24.262859 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 11:13:24.263137 6955 webserver.cc:460] Webserver started at http://127.0.98.61:40187/ using document root <none> and password file <none>
I20260927 11:13:24.263247 6955 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:24.263300 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:24.263856 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.264323 7072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.264434 6955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.264477 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/wal
uuid: "bfc088fef7a4465e8a6176585e798392"
format_stamp: "Formatted at 2026-09-27 11:13:24 on dist-test-slave-9hq9"
server_key: "927ceba1c083985bd879ad3f7fc2fdd9"
server_key_iv: "1bfb58f558064263cf14fcfb0bd723cb"
server_key_version: "encryptionkey@0"
I20260927 11:13:24.264530 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:24.287460 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:24.288995 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:24.292120 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40661
I20260927 11:13:24.292287 7124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40661 every 8 connection(s)
I20260927 11:13:24.292526 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-1/data/info.pb
I20260927 11:13:24.293120 7125 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:13:24.293504 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: Bootstrap starting.
I20260927 11:13:24.295584 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:13:24.295806 7125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: Bootstrap complete.
I20260927 11:13:24.295987 7125 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: true } }
I20260927 11:13:24.296054 7125 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bfc088fef7a4465e8a6176585e798392, State: Initialized, Role: LEARNER
I20260927 11:13:24.296093 7125 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: true } }
I20260927 11:13:24.296262 7125 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:24.296252 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: true } } }
I20260927 11:13:24.296347 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: This master's current role is: LEARNER
I20260927 11:13:24.297668 7139 catalog_manager.cc:1259] Loaded cluster ID: b56755796adc430a96201df52bbd5158
I20260927 11:13:24.297712 7139 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: loading cluster ID for follower catalog manager: success
I20260927 11:13:24.298803 7139 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: acquiring CA information for follower catalog manager: success
I20260927 11:13:24.299057 7139 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 11:13:24.305495 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.298388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.0.98.61:40661)
0927 11:13:24.299154 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.299156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.299165 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:13:24.299191 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.299191 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.299199 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.299254 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.299905 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.300473 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.301487 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.301664 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.301706 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.303095 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.303096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.303096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.303101 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.304458 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.304823 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.304823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.304823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.304870 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.305071 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.305071 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.305072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.305153 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.305234 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.305377 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.305430 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":44,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 11:13:24.328775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41863
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45085,127.0.98.61:40661,127.0.98.60:41863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:24.406767 7143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:24.406937 7143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:24.406957 7143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:24.408056 7143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:24.408094 7143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:24.408108 7143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:24.408157 7143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:24.408174 7143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:24.409605 7143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45085,127.0.98.61:40661,127.0.98.60:41863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:24.409834 7143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:24.410063 7143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:24.414654 7151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:24.414830 7148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:24.414817 7149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:24.415028 7143 server_base.cc:1034] running on GCE node
I20260927 11:13:24.415227 7143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:24.415470 7143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:24.416627 7143 hybrid_clock.cc:648] HybridClock initialized: now 1790507604416621 us; error 37 us; skew 500 ppm
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:24.418921 7143 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 11:13:24.419298 7143 webserver.cc:460] Webserver started at http://127.0.98.60:39161/ using document root <none> and password file <none>
I20260927 11:13:24.419521 7143 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:24.419586 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:24.419728 7143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:24.420658 7143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/data/instance:
uuid: "ca29eb8ceffc4dbb8dee1345574da3cb"
format_stamp: "Formatted at 2026-09-27 11:13:24 on dist-test-slave-9hq9"
server_key: "d40be9022a44b7cbf0a35b7127f434ea"
server_key_iv: "79c3f7ff59626c073a31ae9529686545"
server_key_version: "encryptionkey@0"
I20260927 11:13:24.421017 7143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/wal/instance:
uuid: "ca29eb8ceffc4dbb8dee1345574da3cb"
format_stamp: "Formatted at 2026-09-27 11:13:24 on dist-test-slave-9hq9"
server_key: "d40be9022a44b7cbf0a35b7127f434ea"
server_key_iv: "79c3f7ff59626c073a31ae9529686545"
server_key_version: "encryptionkey@0"
I20260927 11:13:24.422236 7143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:13:24.422892 7158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.423097 7143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:24.423188 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/wal
uuid: "ca29eb8ceffc4dbb8dee1345574da3cb"
format_stamp: "Formatted at 2026-09-27 11:13:24 on dist-test-slave-9hq9"
server_key: "d40be9022a44b7cbf0a35b7127f434ea"
server_key_iv: "79c3f7ff59626c073a31ae9529686545"
server_key_version: "encryptionkey@0"
I20260927 11:13:24.423264 7143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:24.430754 7143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:24.432864 7143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:24.433020 7143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:24.437242 7143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41863
I20260927 11:13:24.437273 7210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41863 every 8 connection(s)
I20260927 11:13:24.437600 7143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/data/info.pb
I20260927 11:13:24.438583 7211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:24.439996 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:13:24.443017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 7143
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 11:13:24.449038 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.440779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36276 (local address 127.0.98.62:45085)
0927 11:13:24.440821 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.440823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.440895 (+ 72us) server_negotiation.cc:407] Connection header received
0927 11:13:24.441326 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.441327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.441337 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.441393 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.441902 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.442436 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.443664 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.443840 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.443916 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.446301 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.446302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.446302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.446308 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.447653 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.448137 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.448138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.448138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.448183 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.448436 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.448436 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.448437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.448537 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.448616 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.448882 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.448953 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:13:24.449209 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.440641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:36276)
0927 11:13:24.440787 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.440825 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.441244 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.441459 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.441462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.441671 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.441836 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.441840 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.442504 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.442504 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.443590 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.443593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.443920 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.443921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.444058 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.444572 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.444581 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.446227 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.447730 (+ 1503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.447736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.447738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.448047 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.448244 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.448245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.448245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.448366 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.448608 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.448614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.448745 (+ 131us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.448845 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.448983 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260927 11:13:24.449747 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:13:24.452250 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.443485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55772 (local address 127.0.98.60:41863)
0927 11:13:24.443636 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.443640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.443659 (+ 19us) server_negotiation.cc:407] Connection header received
0927 11:13:24.443679 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.443681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.443707 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.443788 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.444390 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.445411 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.446421 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.446857 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.446932 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.449481 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.449485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.449485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.449491 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.451085 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.451467 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.451470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.451470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.451522 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.451726 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.451727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.451727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.451859 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.452001 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.452075 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.452150 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 11:13:24.456414 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.450103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40661 (local address 127.0.0.1:44724)
0927 11:13:24.450143 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.450152 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.450184 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.450331 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.450332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.450450 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.450486 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.450487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.451082 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.451083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.451572 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.451573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.451954 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.451955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.452028 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.452488 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.452493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.454238 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.455471 (+ 1233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.455472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.455472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.455631 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.455842 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.455842 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.455843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.455895 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.456136 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.456137 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.456175 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.456242 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.456328 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 11:13:24.456424 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.450151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.0.98.61:40661)
0927 11:13:24.450192 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.450194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.450205 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:13:24.450234 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.450235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.450259 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.450316 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.450559 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.450990 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.451665 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.451857 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.451955 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.454319 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.454320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.454321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.454327 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.455379 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.455713 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.455714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.455714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.455768 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.455966 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.455967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.455968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.456071 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.456144 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.456269 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.456343 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:13:24.456880 7211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 11:13:24.462581 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.457093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41863 (local address 127.0.0.1:55784)
0927 11:13:24.457136 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.457146 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.457180 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.457294 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.457294 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.457413 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.457433 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.457434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.458004 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.458005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.458467 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.458468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.458827 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.458828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.458873 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.459154 (+ 281us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.459157 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.460394 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.461826 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.461827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.461827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.461961 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.462172 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.462173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.462173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.462219 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.462394 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.462394 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.462423 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.462472 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.462529 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 11:13:24.462668 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.457154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.0.98.60:41863)
0927 11:13:24.457183 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.457185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.457192 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:13:24.457230 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.457230 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.457239 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.457290 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.457522 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.457923 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.458552 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.458750 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.458827 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.460463 (+ 1636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.460464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.460464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.460470 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.461762 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.462019 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.462019 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.462020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.462104 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.462267 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.462267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.462268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.462347 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.462389 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.462521 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.462579 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:13:24.464025 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: Bootstrap starting.
I20260927 11:13:24.464892 7211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:24.465173 7211 log.cc:824] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:24.465884 7211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: No bootstrap required, opened a new log
I20260927 11:13:24.467211 7211 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } }
I20260927 11:13:24.467334 7211 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:24.467356 7211 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca29eb8ceffc4dbb8dee1345574da3cb, State: Initialized, Role: FOLLOWER
I20260927 11:13:24.467438 7211 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } }
I20260927 11:13:24.467797 7216 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } } }
I20260927 11:13:24.467849 7211 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:24.467932 7216 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [sys.catalog]: This master's current role is: FOLLOWER
W20260927 11:13:24.469560 7227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:13:24.469647 7227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:13:24.472496 7143 master_runner.cc:428] Detected that this master ca29eb8ceffc4dbb8dee1345574da3cb is joining an existing cluster
I20260927 11:13:24.472558 7143 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:41863
I20260927 11:13:24.495484 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.473796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.0.98.62:45085)
0927 11:13:24.473868 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.473870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.473960 (+ 90us) server_negotiation.cc:407] Connection header received
0927 11:13:24.474087 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.474088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.474100 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.474133 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.474560 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.475340 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.476515 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.476804 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.476843 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.478022 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.478023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.478023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.478031 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.479671 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.494519 (+ 14848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.494521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.494521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.494604 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.494868 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.494870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.494870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.495023 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.495069 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.495286 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.495381 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 11:13:24.495843 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.474147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.0.98.60:41863)
0927 11:13:24.474181 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.474183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.474783 (+ 600us) server_negotiation.cc:407] Connection header received
0927 11:13:24.474860 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.474860 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.474867 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.474892 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.475261 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.475959 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.477026 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.477626 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.477670 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.479175 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.479177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.479177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.479184 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.480628 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.494828 (+ 14200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.494829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.494830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.494881 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.495081 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.495081 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.495082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.495194 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.495235 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.495418 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.495491 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:13:24.495841 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.473745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:36286)
0927 11:13:24.473895 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.473905 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.473986 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.474204 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.474205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.474416 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.474473 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.474475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.475439 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.475440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.476414 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.476416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.476928 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.476929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.477013 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.477470 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.477475 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.477931 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.479754 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.479755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.479756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.494409 (+ 14653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.494689 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.494690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.494691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.494764 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.495117 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.495117 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.495174 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.495616 (+ 442us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.495722 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":76,"thread_start_us":88,"threads_started":1}
I20260927 11:13:24.496110 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.474217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41863 (local address 127.0.0.1:55786)
0927 11:13:24.474734 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.474743 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.474801 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.474935 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.474936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.475124 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.475187 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.475189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.476045 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.476046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.476936 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.476938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.477785 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.477786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.477844 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.478731 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.478736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.479088 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.480704 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.480705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.480705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.494728 (+ 14023us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.494950 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.494951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.494951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.495013 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.495278 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.495278 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.495316 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.495855 (+ 539us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.495956 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":63,"threads_started":1}
I20260927 11:13:24.498881 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.473786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44732 (local address 127.0.98.61:40661)
0927 11:13:24.473827 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.473829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.474147 (+ 318us) server_negotiation.cc:407] Connection header received
0927 11:13:24.474257 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.474258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.474267 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.474296 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.474706 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.475413 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.476507 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.476796 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.476853 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.495377 (+ 18524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.495379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.495379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.495388 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.497722 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.498139 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.498140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.498141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.498198 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.498381 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.498381 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.498382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.498493 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.498570 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.498726 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.498800 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:13:24.498888 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.473907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40661 (local address 127.0.0.1:44732)
0927 11:13:24.474081 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.474091 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.474184 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.474362 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.474363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.474567 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.474627 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.474629 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.475494 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.475495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.476414 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.476416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.476923 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.476924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.477261 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.494887 (+ 17626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.494892 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.495279 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.497801 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.497802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.497802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.498071 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.498268 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.498269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.498269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.498327 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.498562 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.498563 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.498602 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.498689 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.498788 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":54,"thread_start_us":89,"threads_started":1}
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507603, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 11:13:24.503801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 6553
I20260927 11:13:24.504840 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.497965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41863 (local address 127.0.0.1:55788)
0927 11:13:24.498013 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.498024 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.498080 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.498312 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.498313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.498477 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.498524 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.498527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.499416 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.499417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.499924 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.499926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.500374 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.500375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.500419 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.500838 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.500842 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.502197 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.503757 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.503758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.503758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.503970 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.504170 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.504171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.504171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.504221 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.504457 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.504458 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.504514 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.504592 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.504696 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 11:13:24.504866 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.498153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55788 (local address 127.0.98.60:41863)
0927 11:13:24.498188 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.498190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.498196 (+ 6us) server_negotiation.cc:407] Connection header received
0927 11:13:24.498225 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.498227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.498237 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.498291 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.498613 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.499340 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.500002 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.500300 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.500372 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.502270 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.502271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.502272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.502278 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.503665 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.504045 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.504046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.504047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.504095 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.504280 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.504280 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.504281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.504386 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.504448 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.504643 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.504709 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:13:24.505254 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:41863: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 41863
}
attrs {
promote: true
}
}
dest_uuid: "e3930554fc654a9eb440e352ed32c2df"
cas_config_opid_index: 5
I20260927 11:13:24.505518 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: true } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } }
I20260927 11:13:24.505975 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ca29eb8ceffc4dbb8dee1345574da3cb (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: true } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } } }
I20260927 11:13:24.507151 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:41863
I20260927 11:13:24.507212 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: true } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } } }
I20260927 11:13:24.507265 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:24.509549 7143 master.cc:537] Master@127.0.98.60:41863 shutting down...
I20260927 11:13:24.510208 7185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:24.511276 7185 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 1 FOLLOWER]: Refusing update from remote peer e3930554fc654a9eb440e352ed32c2df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 11:13:24.513010 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.506213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40661 (local address 127.0.0.1:44736)
0927 11:13:24.507552 (+ 1339us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.507563 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.507602 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.509740 (+ 2138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.509742 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.509743 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:24.509780 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.509782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.510966 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.510967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.512127 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.512129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.512743 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.512744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.512799 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.512818 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.512875 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.512935 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1325}
I20260927 11:13:24.513175 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.508578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44736 (local address 127.0.98.61:40661)
0927 11:13:24.509557 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.509559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.509567 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:13:24.509591 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.509591 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.509612 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.509678 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:24.509848 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.510613 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.512222 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.512656 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.512742 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.512812 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.512881 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.512920 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956}
I20260927 11:13:24.514428 7143 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:24.514513 7143 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:24.514559 7143 tablet_replica.cc:330] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: stopping tablet replica
I20260927 11:13:24.514724 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:24.514770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36085
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45085
--webserver_port=40783
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45085,127.0.98.61:40661
--tserver_master_addrs=127.0.98.62:45085,127.0.98.61:40661,127.0.98.60:41863 with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:13:24.515062 7099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 LEARNER]: Refusing update from remote peer e3930554fc654a9eb440e352ed32c2df: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 11:13:24.515244 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260927 11:13:24.515381 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df -> Peer ca29eb8ceffc4dbb8dee1345574da3cb (127.0.98.60:41863): Couldn't send request to peer ca29eb8ceffc4dbb8dee1345574da3cb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 11:13:24.515844 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ca29eb8ceffc4dbb8dee1345574da3cb (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: true } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } } }
I20260927 11:13:24.517222 7051 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df: attempting to promote NON_VOTER bfc088fef7a4465e8a6176585e798392 to VOTER
I20260927 11:13:24.517246 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3930554fc654a9eb440e352ed32c2df. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: true } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } } }
I20260927 11:13:24.517326 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: This master's current role is: LEARNER
I20260927 11:13:24.517364 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: true } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } } }
I20260927 11:13:24.517401 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: This master's current role is: LEARNER
I20260927 11:13:24.517395 7051 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } }
I20260927 11:13:24.517884 7099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 LEARNER]: Refusing update from remote peer e3930554fc654a9eb440e352ed32c2df: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260927 11:13:24.518356 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df -> Peer ca29eb8ceffc4dbb8dee1345574da3cb (127.0.98.60:41863): Couldn't send request to peer ca29eb8ceffc4dbb8dee1345574da3cb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 11:13:24.518503 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 11:13:24.522462 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bfc088fef7a4465e8a6176585e798392 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } } }
I20260927 11:13:24.523343 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } } }
I20260927 11:13:24.523420 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:24.523466 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } } }
I20260927 11:13:24.523518 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:24.523779 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bfc088fef7a4465e8a6176585e798392 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } } }
I20260927 11:13:24.524606 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } } }
I20260927 11:13:24.524690 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:13:24.529857 7143 master.cc:552] Master@127.0.98.60:41863 shutdown complete.
I20260927 11:13:24.529915 7143 master_runner.cc:305] Clearing existing system tablet
I20260927 11:13:24.530105 7143 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:24.530159 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:24.530966 7143 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:24.531495 7241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.531605 7143 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:24.531651 7143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.531673 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/wal
uuid: "ca29eb8ceffc4dbb8dee1345574da3cb"
format_stamp: "Formatted at 2026-09-27 11:13:24 on dist-test-slave-9hq9"
server_key: "d40be9022a44b7cbf0a35b7127f434ea"
server_key_iv: "79c3f7ff59626c073a31ae9529686545"
server_key_version: "encryptionkey@0"
I20260927 11:13:24.532423 7143 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:24.533764 7143 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 11:13:24.533841 7143 log.cc:1196] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 11:13:24.534034 7143 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: Deleting consensus metadata
I20260927 11:13:24.534178 7143 master_runner.cc:315] Copying system tablet from 127.0.98.62:45085
I20260927 11:13:24.534739 7143 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45085
I20260927 11:13:24.542155 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.535082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36300 (local address 127.0.98.62:45085)
0927 11:13:24.535123 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.535125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.535182 (+ 57us) server_negotiation.cc:407] Connection header received
0927 11:13:24.535244 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.535245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.535254 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.535323 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.535663 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.536439 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.537837 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.538297 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.538371 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.539413 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.539415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.539415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.539422 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.541020 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.541388 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.541389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.541390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.541445 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.541639 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.541639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.541640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.541757 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.541819 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.541981 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.542078 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:13:24.542163 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.535020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:36300)
0927 11:13:24.535130 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.535140 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.535194 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.535329 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.535330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.535516 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.535590 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.535592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.536525 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.536526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.537738 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.537741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.538373 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.538375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.538439 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.538954 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.538959 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.539321 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.541100 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.541101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.541102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.541312 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.541512 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.541513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.541513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.541576 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.541821 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.541821 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.541866 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.541937 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.542036 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":55,"threads_started":1}
I20260927 11:13:24.542743 6511 tablet_copy_service.cc:140] P e3930554fc654a9eb440e352ed32c2df: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ca29eb8ceffc4dbb8dee1345574da3cb ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36300)
I20260927 11:13:24.542804 6511 tablet_copy_service.cc:161] P e3930554fc654a9eb440e352ed32c2df: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ca29eb8ceffc4dbb8dee1345574da3cb at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36300: session id = ca29eb8ceffc4dbb8dee1345574da3cb-00000000000000000000000000000000
I20260927 11:13:24.543345 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 11:13:24.543746 7143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 11:13:24.544809 7143 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: tablet copy: Starting download of 0 data blocks...
I20260927 11:13:24.544970 7143 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: tablet copy: Starting download of 1 WAL segments...
I20260927 11:13:24.545655 7143 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 11:13:24.546468 6511 tablet_copy_service.cc:342] P e3930554fc654a9eb440e352ed32c2df: Request end of tablet copy session ca29eb8ceffc4dbb8dee1345574da3cb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:36300
I20260927 11:13:24.546520 6511 tablet_copy_service.cc:433] P e3930554fc654a9eb440e352ed32c2df: ending tablet copy session ca29eb8ceffc4dbb8dee1345574da3cb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ca29eb8ceffc4dbb8dee1345574da3cb
I20260927 11:13:24.549386 7143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:24.550419 7248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:24.550495 7251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:24.550689 7249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:24.554524 7143 server_base.cc:1034] running on GCE node
I20260927 11:13:24.554613 7143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:24.554733 7143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:24.555876 7143 hybrid_clock.cc:648] HybridClock initialized: now 1790507604555855 us; error 42 us; skew 500 ppm
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:24.557268 7143 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 11:13:24.557514 7143 webserver.cc:460] Webserver started at http://127.0.98.60:40547/ using document root <none> and password file <none>
I20260927 11:13:24.557626 7143 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:24.557662 7143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:24.558347 7143 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:24.558924 7258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.559082 7143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:24.559132 7143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/wal
uuid: "ca29eb8ceffc4dbb8dee1345574da3cb"
format_stamp: "Formatted at 2026-09-27 11:13:24 on dist-test-slave-9hq9"
server_key: "d40be9022a44b7cbf0a35b7127f434ea"
server_key_iv: "79c3f7ff59626c073a31ae9529686545"
server_key_version: "encryptionkey@0"
I20260927 11:13:24.559178 7143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:24.574146 7143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:24.576013 7143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:24.579161 7143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41863
I20260927 11:13:24.579562 7143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/master-2/data/info.pb
I20260927 11:13:24.580011 7310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41863 every 8 connection(s)
I20260927 11:13:24.580425 7311 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:13:24.580891 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: Bootstrap starting.
I20260927 11:13:24.584749 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:13:24.584962 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: Bootstrap complete.
I20260927 11:13:24.585150 7311 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } }
I20260927 11:13:24.585222 7311 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ca29eb8ceffc4dbb8dee1345574da3cb, State: Initialized, Role: LEARNER
I20260927 11:13:24.585253 7311 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } }
I20260927 11:13:24.585453 7311 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:24.586135 7314 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } } }
I20260927 11:13:24.586375 7314 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [sys.catalog]: This master's current role is: LEARNER
I20260927 11:13:24.586964 7325 catalog_manager.cc:1259] Loaded cluster ID: b56755796adc430a96201df52bbd5158
I20260927 11:13:24.589502 7325 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: loading cluster ID for follower catalog manager: success
I20260927 11:13:24.590864 7325 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: acquiring CA information for follower catalog manager: success
I20260927 11:13:24.591156 7325 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 11:13:24.621527 7237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:24.621695 7237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:24.621716 7237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:24.622841 7237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:24.622880 7237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:24.622906 7237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:24.624226 7237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40783
--tserver_master_addrs=127.0.98.62:45085,127.0.98.61:40661,127.0.98.60:41863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:24.624383 7237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:24.624547 7237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:24.629942 7331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:24.630282 7330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:24.629966 7333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:24.631878 7237 server_base.cc:1034] running on GCE node
I20260927 11:13:24.632091 7237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:24.632309 7237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:24.633464 7237 hybrid_clock.cc:648] HybridClock initialized: now 1790507604633443 us; error 39 us; skew 500 ppm
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:24.635995 7237 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 11:13:24.636371 7237 webserver.cc:460] Webserver started at http://127.0.98.1:40783/ using document root <none> and password file <none>
I20260927 11:13:24.636577 7237 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:24.636663 7237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:24.637948 7237 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:24.638572 7340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.638736 7237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.638827 7237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "ff5b9fa623da4e6cb3389c7f86669d63"
format_stamp: "Formatted at 2026-09-27 11:13:23 on dist-test-slave-9hq9"
server_key: "690e508531e3a08c67bb75b2bd5155c2"
server_key_iv: "19eee59ec649247f7e6b365adb6c9a58"
server_key_version: "encryptionkey@0"
I20260927 11:13:24.639165 7237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:24.652252 7237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:24.654021 7237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:24.654198 7237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:24.654399 7237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:24.654690 7237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:24.654742 7237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.654781 7237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:24.654809 7237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.660424 7237 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36085
I20260927 11:13:24.660458 7453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36085 every 8 connection(s)
I20260927 11:13:24.660779 7237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:13:24.663136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 7237
I20260927 11:13:24.663283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 6687
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 11:13:24.670099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34545
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45085
--webserver_port=45829
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45085,127.0.98.61:40661
--tserver_master_addrs=127.0.98.62:45085,127.0.98.61:40661,127.0.98.60:41863 with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:13:24.673024 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.661819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49841 (local address 127.0.98.62:45085)
0927 11:13:24.661872 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.661876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.662115 (+ 239us) server_negotiation.cc:407] Connection header received
0927 11:13:24.663315 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.663317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.663330 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.663362 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.664705 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.665544 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.666566 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.666772 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.666856 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.669550 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.669551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.669552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.669560 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.671512 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.671979 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.671980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.671981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.672041 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.672292 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.672292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.672293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.672418 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.672459 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.672830 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.672926 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 11:13:24.673228 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.661823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.1:49841)
0927 11:13:24.661971 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.662011 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.663181 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.663520 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.663524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.664406 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.664607 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.664613 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.665642 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.665643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.666465 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.666467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.666864 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.666865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.666988 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.667494 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.667503 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.669448 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.671614 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.671620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.671622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.671887 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.672111 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.672112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.672112 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.672225 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.672502 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.672507 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.672676 (+ 169us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.672809 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.672976 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":9,"thread_start_us":94,"threads_started":1}
I20260927 11:13:24.673448 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.661825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41863 (local address 127.0.98.1:60465)
0927 11:13:24.661973 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.662011 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.663194 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.663525 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.663526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.664061 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.664697 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.664698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.666021 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.666022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.667055 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.667057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.667482 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.667483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.667557 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.667991 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.667997 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.670096 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.672269 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.672270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.672271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.672434 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.672635 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.672636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.672637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.672705 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.673127 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.673128 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.673179 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.673239 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.673321 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260927 11:13:24.673666 7455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45085
I20260927 11:13:24.673753 7455 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:24.673815 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.662088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60465 (local address 127.0.98.60:41863)
0927 11:13:24.662176 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.662178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.662188 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:13:24.663301 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.663302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.663315 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.663353 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.664772 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.665925 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.667142 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.667412 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.667490 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.670190 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.670191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.670192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.670200 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.672174 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.672517 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.672518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.672518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.672572 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.672904 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.672905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.672905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.673035 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.673097 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.673620 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.673704 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":44,"threads_started":1}
I20260927 11:13:24.673955 7455 heartbeater.cc:511] Master 127.0.98.62:45085 requested a full tablet report, sending...
I20260927 11:13:24.674175 7454 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41863
I20260927 11:13:24.674217 7454 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:24.674285 7454 heartbeater.cc:511] Master 127.0.98.60:41863 requested a full tablet report, sending...
I20260927 11:13:24.674355 6491 ts_manager.cc:194] Re-registered known tserver with Master: ff5b9fa623da4e6cb3389c7f86669d63 (127.0.98.1:36085)
I20260927 11:13:24.674574 7266 ts_manager.cc:194] Registered new tserver with Master: ff5b9fa623da4e6cb3389c7f86669d63 (127.0.98.1:36085)
I20260927 11:13:24.674801 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49841
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 11:13:24.678236 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.662205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39589 (local address 127.0.98.61:40661)
0927 11:13:24.662251 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.662253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.662311 (+ 58us) server_negotiation.cc:407] Connection header received
0927 11:13:24.663301 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.663302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.663315 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.663349 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.669248 (+ 5899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.669696 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.672020 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.672340 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.672383 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.675689 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.675690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.675690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.675696 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.677069 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.677453 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.677454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.677454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.677500 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.677705 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.677705 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.677705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.677804 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.677852 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.678035 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.678178 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 11:13:24.678301 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.662123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40661 (local address 127.0.98.1:39589)
0927 11:13:24.662257 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.662266 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.663194 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.663525 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.663526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.664473 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.664606 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.664612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.669781 (+ 5169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.669783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.671926 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.671928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.672459 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.672460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.672528 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.673721 (+ 1193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.673725 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.675596 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.677148 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.677149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.677150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.677377 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.677568 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.677569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.677570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.677640 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.677868 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.677869 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.677916 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.678004 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.678139 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260927 11:13:24.678778 7456 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40661
I20260927 11:13:24.678833 7456 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:24.678905 7456 heartbeater.cc:511] Master 127.0.98.61:40661 requested a full tablet report, sending...
I20260927 11:13:24.679273 7089 ts_manager.cc:194] Registered new tserver with Master: ff5b9fa623da4e6cb3389c7f86669d63 (127.0.98.1:36085)
W20260927 11:13:24.752806 7462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:24.753000 7462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:24.753023 7462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:24.754089 7462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:24.754138 7462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:24.754166 7462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:24.755465 7462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34545
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45829
--tserver_master_addrs=127.0.98.62:45085,127.0.98.61:40661,127.0.98.60:41863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:24.755641 7462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:24.755786 7462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:24.760309 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:24.760337 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:24.760501 7462 server_base.cc:1034] running on GCE node
W20260927 11:13:24.760586 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:24.760792 7462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:24.761029 7462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:24.762179 7462 hybrid_clock.cc:648] HybridClock initialized: now 1790507604762178 us; error 37 us; skew 500 ppm
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:24.764559 7462 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 11:13:24.765056 7462 webserver.cc:460] Webserver started at http://127.0.98.2:45829/ using document root <none> and password file <none>
I20260927 11:13:24.765275 7462 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:24.765354 7462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:24.766716 7462 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:13:24.767257 7477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.767431 7462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:24.767509 7462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "528b23b21f864e4dacae07daace39679"
format_stamp: "Formatted at 2026-09-27 11:13:23 on dist-test-slave-9hq9"
server_key: "d964891fc48dd99319d9aade7c06be43"
server_key_iv: "c627edce7656f80c6f38bd03b549aee1"
server_key_version: "encryptionkey@0"
I20260927 11:13:24.767782 7462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:24.791517 7462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:24.793357 7462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:24.793494 7462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:24.793708 7462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:24.793983 7462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:24.794015 7462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.794083 7462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:24.794131 7462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.799899 7462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34545
I20260927 11:13:24.799942 7590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34545 every 8 connection(s)
I20260927 11:13:24.800261 7462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:13:24.807545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 7462
I20260927 11:13:24.807717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 6821
I20260927 11:13:24.811128 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.801328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60447 (local address 127.0.98.62:45085)
0927 11:13:24.801391 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.801393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.801494 (+ 101us) server_negotiation.cc:407] Connection header received
0927 11:13:24.801698 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.801699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.801724 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.801756 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.802535 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.803270 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.804381 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.804674 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.804764 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.808034 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.808036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.808036 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.808044 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.809591 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.810002 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.810004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.810005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.810101 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.810386 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.810388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.810388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.810549 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.810621 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.810906 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.810991 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:13:24.811183 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.800859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47833 (local address 127.0.98.60:41863)
0927 11:13:24.800915 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.800917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.801114 (+ 197us) server_negotiation.cc:407] Connection header received
0927 11:13:24.801699 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.801700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.801724 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.801771 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.802510 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.803011 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.803957 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.804184 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.804263 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.807664 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.807666 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.807666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.807673 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.810055 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.810432 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.810433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.810434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.810484 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.810687 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.810688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.810689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.810803 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.810845 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.811025 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.811100 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 11:13:24.811292 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.801279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.2:60447)
0927 11:13:24.801435 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.801442 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.801609 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.801856 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.801856 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.802338 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.802423 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.802429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.803369 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.803369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.804301 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.804303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.804809 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.804810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.804914 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.805435 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.805441 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.807924 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.809693 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.809699 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.809701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.809914 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.810171 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.810172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.810172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.810308 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.810608 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.810613 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.810762 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.810867 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.811038 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":29,"thread_start_us":70,"threads_started":1}
I20260927 11:13:24.811626 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.800874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41863 (local address 127.0.98.2:47833)
0927 11:13:24.801004 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.801041 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.801592 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.801851 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.801855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.802248 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.802422 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.802426 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.803113 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.803114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.803858 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.803860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.804272 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.804273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.804413 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.804953 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.804962 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.807573 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.810148 (+ 2575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.810149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.810150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.810347 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.810561 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.810561 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.810562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.810616 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.810890 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.810891 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.810923 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.811493 (+ 570us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.811568 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20260927 11:13:24.811774 7592 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45085
I20260927 11:13:24.811865 7592 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:24.811936 7591 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41863
I20260927 11:13:24.811986 7591 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:24.812083 7592 heartbeater.cc:511] Master 127.0.98.62:45085 requested a full tablet report, sending...
I20260927 11:13:24.812085 7591 heartbeater.cc:511] Master 127.0.98.60:41863 requested a full tablet report, sending...
I20260927 11:13:24.812551 7266 ts_manager.cc:194] Registered new tserver with Master: 528b23b21f864e4dacae07daace39679 (127.0.98.2:34545)
I20260927 11:13:24.812565 6491 ts_manager.cc:194] Re-registered known tserver with Master: 528b23b21f864e4dacae07daace39679 (127.0.98.2:34545)
I20260927 11:13:24.813050 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60447
I20260927 11:13:24.815033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38171
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45085
--webserver_port=43771
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45085,127.0.98.61:40661
--tserver_master_addrs=127.0.98.62:45085,127.0.98.61:40661,127.0.98.60:41863 with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:13:24.815614 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.801157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41231 (local address 127.0.98.61:40661)
0927 11:13:24.801198 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.801200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.801301 (+ 101us) server_negotiation.cc:407] Connection header received
0927 11:13:24.801745 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.801746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.801756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.801821 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.802529 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.803281 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.804402 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.804644 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.804701 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.807321 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.807322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.807322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.807329 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.809248 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.814809 (+ 5561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.814810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.814810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.814868 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.815080 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.815080 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.815081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.815224 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.815283 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.815454 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.815533 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:13:24.815769 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.801105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40661 (local address 127.0.98.2:41231)
0927 11:13:24.801238 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.801249 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.801612 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.801860 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.801860 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.802304 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.802423 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.802429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.803369 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.803369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.804303 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.804304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.804752 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.804753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.804813 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.805227 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.805231 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.807216 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.814581 (+ 7365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.814582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.814582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.814727 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.814938 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.814939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.814940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.815001 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.815310 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.815311 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.815350 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.815418 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.815522 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":57,"thread_start_us":52,"threads_started":1}
I20260927 11:13:24.816136 7593 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40661
I20260927 11:13:24.816195 7593 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:24.816260 7593 heartbeater.cc:511] Master 127.0.98.61:40661 requested a full tablet report, sending...
I20260927 11:13:24.816567 7089 ts_manager.cc:194] Registered new tserver with Master: 528b23b21f864e4dacae07daace39679 (127.0.98.2:34545)
W20260927 11:13:24.893556 7598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:24.893728 7598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:24.893749 7598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:24.894901 7598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:24.894941 7598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:24.894997 7598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:24.896497 7598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43771
--tserver_master_addrs=127.0.98.62:45085,127.0.98.61:40661,127.0.98.60:41863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:24.896715 7598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:24.896929 7598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:24.901753 7606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:24.901804 7604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:24.901826 7598 server_base.cc:1034] running on GCE node
W20260927 11:13:24.901804 7603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:24.902181 7598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:24.902438 7598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:24.903589 7598 hybrid_clock.cc:648] HybridClock initialized: now 1790507604903580 us; error 38 us; skew 500 ppm
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:24.906029 7598 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 11:13:24.906461 7598 webserver.cc:460] Webserver started at http://127.0.98.3:43771/ using document root <none> and password file <none>
I20260927 11:13:24.906682 7598 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:24.906756 7598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:24.908021 7598 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:24.908525 7613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.908674 7598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.908764 7598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "24208615a2e54f9d9638016054d89e17"
format_stamp: "Formatted at 2026-09-27 11:13:23 on dist-test-slave-9hq9"
server_key: "644134a097358c70cf2dcb1a781852f0"
server_key_iv: "314747e47df016808fc67e022488568c"
server_key_version: "encryptionkey@0"
I20260927 11:13:24.909083 7598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:24.919162 7598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:24.921283 7598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:24.921429 7598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:24.921624 7598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:24.921887 7598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:24.921937 7598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.921974 7598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:24.922003 7598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:24.927881 7598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38171
I20260927 11:13:24.927973 7726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38171 every 8 connection(s)
I20260927 11:13:24.928258 7598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:13:24.929244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 7598
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 11:13:24.940333 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.930772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.0.98.60:41863)
0927 11:13:24.930991 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.930993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.931141 (+ 148us) server_negotiation.cc:407] Connection header received
0927 11:13:24.931165 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.931166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.931176 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.931228 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.931925 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.932680 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.933846 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.934182 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.934243 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.936212 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.936214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.936214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.936220 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.937857 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.938546 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.938547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.938547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.938606 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.939544 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.939544 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.939545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.939673 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.939763 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.940129 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.940211 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":56,"threads_started":1}
I20260927 11:13:24.941643 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.930628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.0.98.61:40661)
0927 11:13:24.930807 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.930809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.930968 (+ 159us) server_negotiation.cc:407] Connection header received
0927 11:13:24.930994 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.930995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.931007 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.931074 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.931919 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.932673 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.934380 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.934670 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.934708 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.936383 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.936384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.936384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.936390 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.939924 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.940339 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.940340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.940341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.940387 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.940597 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.940598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.940598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.940711 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.940782 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.941005 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.941537 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":57,"threads_started":1}
Sep 27 11:13:24 dist-test-slave-9hq9 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507604, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 11:13:24.942293 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.928936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34785 (local address 127.0.98.62:45085)
0927 11:13:24.928982 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.928984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.929415 (+ 431us) server_negotiation.cc:407] Connection header received
0927 11:13:24.930733 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.930735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.930747 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.930802 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.931804 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.932559 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.934689 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.934988 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.935063 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.937459 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.937460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.937461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.937467 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.939319 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.940919 (+ 1600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.940920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.940921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.940977 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.941283 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.941284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.941284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.941398 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.941457 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.942130 (+ 673us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.942200 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:13:24.942783 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.928938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.3:34785)
0927 11:13:24.929302 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.929311 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.930534 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.930928 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.930932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.931399 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.931703 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.931708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.932668 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.932668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.934599 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.934601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.935115 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.935116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.935192 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.935635 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.935644 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.937367 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.940265 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.940283 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.940285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.940839 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.941071 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.941072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.941072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.941204 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.941483 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.941488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.941634 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.941729 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.942538 (+ 809us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":1009,"thread_start_us":86,"threads_started":1}
I20260927 11:13:24.943197 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45085
I20260927 11:13:24.943297 7728 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:24.943305 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.929215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40661 (local address 127.0.98.3:50373)
0927 11:13:24.929519 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.929528 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.930673 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.930928 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.930932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.931483 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.931703 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.931708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.932663 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.932664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.933909 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.933912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.934404 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.934405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.934802 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.936420 (+ 1618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.936424 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.939401 (+ 2977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.941839 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.941840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.941840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.942099 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.942279 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.942279 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.942280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.942327 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.942950 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.942951 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.943008 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.943059 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.943144 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":5,"thread_start_us":52,"threads_started":1}
I20260927 11:13:24.943496 7728 heartbeater.cc:511] Master 127.0.98.62:45085 requested a full tablet report, sending...
I20260927 11:13:24.943696 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.929359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50373 (local address 127.0.98.61:40661)
0927 11:13:24.929414 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.929416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.929567 (+ 151us) server_negotiation.cc:407] Connection header received
0927 11:13:24.930749 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.930750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.930760 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.930806 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.931803 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.932566 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.933991 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.934293 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.934350 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.939480 (+ 5130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.939481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.939481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.939487 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.941666 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.942170 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.942171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.942172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.942220 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.942525 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.942526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.942527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.942871 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.942925 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.943499 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.943610 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 11:13:24.943853 6491 ts_manager.cc:194] Re-registered known tserver with Master: 24208615a2e54f9d9638016054d89e17 (127.0.98.3:38171)
I20260927 11:13:24.943941 7729 heartbeater.cc:347] Connected to a master server at 127.0.98.61:40661
I20260927 11:13:24.943989 7729 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:24.944053 7729 heartbeater.cc:511] Master 127.0.98.61:40661 requested a full tablet report, sending...
I20260927 11:13:24.944285 7089 ts_manager.cc:194] Registered new tserver with Master: 24208615a2e54f9d9638016054d89e17 (127.0.98.3:38171)
I20260927 11:13:24.944334 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34785
I20260927 11:13:24.945485 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.929171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39259 (local address 127.0.98.60:41863)
0927 11:13:24.929232 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.929234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.929250 (+ 16us) server_negotiation.cc:407] Connection header received
0927 11:13:24.931388 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.931389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.931399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.931430 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.935609 (+ 4179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.936293 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.939260 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.939554 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.939592 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.942436 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.942437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.942437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.942443 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.944083 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.944741 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.944742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.944742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.944794 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.944992 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.944993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.944993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.945097 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.945160 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.945320 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.945402 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 11:13:24.945628 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.928938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41863 (local address 127.0.98.3:39259)
0927 11:13:24.929127 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.929186 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.930986 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.935218 (+ 4232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.935219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.935371 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:24.935430 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.935432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.938290 (+ 2858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.938291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.939127 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.939129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.939931 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.939932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.940006 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.940962 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:24.940967 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:24.942344 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:24.944147 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.944148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.944149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.944655 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.944862 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:24.944862 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:24.944863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:24.944916 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:24.945164 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:24.945165 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:24.945202 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.945274 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.945398 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":101,"threads_started":1}
I20260927 11:13:24.945962 7727 heartbeater.cc:347] Connected to a master server at 127.0.98.60:41863
I20260927 11:13:24.946036 7727 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:24.946131 7727 heartbeater.cc:511] Master 127.0.98.60:41863 requested a full tablet report, sending...
I20260927 11:13:24.946393 7266 ts_manager.cc:194] Registered new tserver with Master: 24208615a2e54f9d9638016054d89e17 (127.0.98.3:38171)
I20260927 11:13:24.946720 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.930476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36304 (local address 127.0.98.62:45085)
0927 11:13:24.932923 (+ 2447us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.932925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.932933 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:13:24.932951 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.932952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.932963 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.933022 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:24.933561 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.936812 (+ 3251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.937840 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.938166 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.938228 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.939644 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:24.939645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:24.939646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:24.939651 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:24.945487 (+ 5836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.945935 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.945935 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.945936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.945988 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:24.946206 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:24.946207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:24.946207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:24.946324 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:24.946403 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.946559 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.946617 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2422,"thread_start_us":81,"threads_started":1}
I20260927 11:13:24.947932 6491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:13:24.949321 6491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:24.956215 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.952367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34545 (local address 127.0.0.1:50306)
0927 11:13:24.952633 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.952644 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.952688 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.952856 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.952856 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.952858 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:24.952895 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.952896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.954153 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.954154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.955206 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.955208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.956010 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.956011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.956063 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.956066 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.956128 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.956164 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":71,"threads_started":1}
I20260927 11:13:24.956259 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.952455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50306 (local address 127.0.98.2:34545)
0927 11:13:24.952603 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.952607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.952687 (+ 80us) server_negotiation.cc:407] Connection header received
0927 11:13:24.952758 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.952759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.952772 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.952837 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:24.952982 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.954087 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.955292 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.955913 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.955993 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.956058 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.956141 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.956179 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260927 11:13:24.956888 7525 tablet_service.cc:1467] Processing CreateTablet for tablet 867a9f1351b6466a9606ea1d99c50ab8 (DEFAULT_TABLE table=test-table [id=251a080382ab49cdafe8abb7539e2277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:24.957103 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.952541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38171 (local address 127.0.0.1:56068)
0927 11:13:24.953208 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.953216 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.953327 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.953501 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.953502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.953504 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:24.953551 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.953553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.954489 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.954490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.955570 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.955572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.956871 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.956872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.956933 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.956938 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.957002 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.957041 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":58,"threads_started":1}
I20260927 11:13:24.957131 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.952987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.0.98.3:38171)
0927 11:13:24.953129 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.953133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.953300 (+ 167us) server_negotiation.cc:407] Connection header received
0927 11:13:24.953395 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.953397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.953410 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.953495 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:24.953601 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.954403 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.955651 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.956780 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.956879 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.956944 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.957022 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.957057 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260927 11:13:24.957243 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867a9f1351b6466a9606ea1d99c50ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:24.957690 7661 tablet_service.cc:1467] Processing CreateTablet for tablet 867a9f1351b6466a9606ea1d99c50ab8 (DEFAULT_TABLE table=test-table [id=251a080382ab49cdafe8abb7539e2277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:24.957965 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867a9f1351b6466a9606ea1d99c50ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:24.960062 7753 tablet_bootstrap.cc:492] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679: Bootstrap starting.
I20260927 11:13:24.960304 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.952337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36085 (local address 127.0.0.1:54758)
0927 11:13:24.952633 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:24.952644 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:24.952688 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:24.953261 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:24.953262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:24.953265 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:24.953301 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.953302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.954432 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.954433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.955260 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:24.955280 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.960066 (+ 4786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.960067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:24.960132 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.960136 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:24.960189 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 11:13:24.960231 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":25}
I20260927 11:13:24.960618 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:24.952796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54758 (local address 127.0.98.1:36085)
0927 11:13:24.952980 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:24.952984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:24.952993 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:13:24.953037 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:24.953039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:24.953163 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:24.953365 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:24.953421 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.954355 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.955344 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:24.955930 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:24.956021 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:24.956094 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:24.960467 (+ 4373us) server_negotiation.cc:299] Negotiation successful
0927 11:13:24.960519 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260927 11:13:24.960906 7754 tablet_bootstrap.cc:492] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17: Bootstrap starting.
I20260927 11:13:24.960990 7753 tablet_bootstrap.cc:654] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:24.961269 7753 log.cc:824] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:24.961344 7754 tablet_bootstrap.cc:654] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:24.961238 7388 tablet_service.cc:1467] Processing CreateTablet for tablet 867a9f1351b6466a9606ea1d99c50ab8 (DEFAULT_TABLE table=test-table [id=251a080382ab49cdafe8abb7539e2277]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:24.961537 7388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 867a9f1351b6466a9606ea1d99c50ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:24.961627 7754 log.cc:824] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:24.962003 7753 tablet_bootstrap.cc:492] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679: No bootstrap required, opened a new log
I20260927 11:13:24.962087 7753 ts_tablet_manager.cc:1397] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:13:24.962324 7754 tablet_bootstrap.cc:492] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17: No bootstrap required, opened a new log
I20260927 11:13:24.962404 7754 ts_tablet_manager.cc:1397] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:24.964469 7754 raft_consensus.cc:348] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } } peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } }
I20260927 11:13:24.964648 7754 raft_consensus.cc:374] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:24.964689 7754 raft_consensus.cc:729] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24208615a2e54f9d9638016054d89e17, State: Initialized, Role: FOLLOWER
I20260927 11:13:24.964797 7754 consensus_queue.cc:260] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } } peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } }
I20260927 11:13:24.964905 7753 raft_consensus.cc:348] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } } peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } }
I20260927 11:13:24.965049 7754 ts_tablet_manager.cc:1428] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 11:13:24.965065 7753 raft_consensus.cc:374] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:24.965103 7753 raft_consensus.cc:729] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528b23b21f864e4dacae07daace39679, State: Initialized, Role: FOLLOWER
I20260927 11:13:24.965173 7728 heartbeater.cc:503] Master 127.0.98.62:45085 was elected leader, sending a full tablet report...
I20260927 11:13:24.965241 7753 consensus_queue.cc:260] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } } peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } }
I20260927 11:13:24.965528 7758 tablet_bootstrap.cc:492] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63: Bootstrap starting.
I20260927 11:13:24.965790 7753 ts_tablet_manager.cc:1428] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 11:13:24.966102 7592 heartbeater.cc:503] Master 127.0.98.62:45085 was elected leader, sending a full tablet report...
I20260927 11:13:24.966259 7758 tablet_bootstrap.cc:654] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:24.966586 7758 log.cc:824] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:24.967299 7758 tablet_bootstrap.cc:492] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63: No bootstrap required, opened a new log
I20260927 11:13:24.967361 7758 ts_tablet_manager.cc:1397] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:13:24.969179 7758 raft_consensus.cc:348] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } } peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } }
I20260927 11:13:24.969292 7758 raft_consensus.cc:374] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:24.969312 7758 raft_consensus.cc:729] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5b9fa623da4e6cb3389c7f86669d63, State: Initialized, Role: FOLLOWER
I20260927 11:13:24.969417 7758 consensus_queue.cc:260] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } } peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } }
I20260927 11:13:24.969666 7758 ts_tablet_manager.cc:1428] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:13:24.969702 7455 heartbeater.cc:503] Master 127.0.98.62:45085 was elected leader, sending a full tablet report...
W20260927 11:13:25.050912 7594 tablet.cc:2369] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:13:25.094976 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:25.092425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55802 (local address 127.0.98.60:41863)
0927 11:13:25.092489 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:25.092491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:25.092501 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:13:25.092568 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:25.092570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:25.092593 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:25.092665 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:25.092797 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.093275 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.094216 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.094647 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.094737 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:25.094839 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:25.094869 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 11:13:25.094902 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 11:13:25.094980 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:25.092349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41863 (local address 127.0.0.1:55802)
0927 11:13:25.092411 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:25.092428 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:25.092492 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:25.092666 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:25.092667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:25.092668 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:25.092727 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:25.092729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.093355 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.093356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:25.094125 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:25.094127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.094738 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.094739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:25.094795 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:25.094799 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:25.094854 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 11:13:25.094901 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 11:13:25.095361 7285 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260927 11:13:25.096231 7762 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3930554fc654a9eb440e352ed32c2df. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } } }
I20260927 11:13:25.096367 7762 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [sys.catalog]: This master's current role is: LEARNER
W20260927 11:13:25.162618 7457 tablet.cc:2369] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:13:25.179726 7730 tablet.cc:2369] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:13:25.294571 7761 raft_consensus.cc:482] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:25.294690 7761 raft_consensus.cc:504] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } } peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } }
I20260927 11:13:25.295001 7761 leader_election.cc:288] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 528b23b21f864e4dacae07daace39679 (127.0.98.2:34545), 24208615a2e54f9d9638016054d89e17 (127.0.98.3:38171)
I20260927 11:13:25.297871 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:25.295234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36201 (local address 127.0.98.2:34545)
0927 11:13:25.295288 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:25.295290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:25.295326 (+ 36us) server_negotiation.cc:407] Connection header received
0927 11:13:25.295383 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:25.295384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:25.295393 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:25.295466 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:25.295610 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.296086 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.297067 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.297547 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.297620 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:25.297630 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:25.297777 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 11:13:25.297812 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 11:13:25.297887 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:25.295162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34545 (local address 127.0.98.1:36201)
0927 11:13:25.295264 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:25.295276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:25.295326 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:25.295480 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:25.295482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:25.295484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:25.295535 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:25.295537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.296167 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.296168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:25.296979 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:25.296981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.297630 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.297631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:25.297690 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:25.297693 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:25.297769 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 11:13:25.297808 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":60,"threads_started":1}
I20260927 11:13:25.298164 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:25.295400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39987 (local address 127.0.98.3:38171)
0927 11:13:25.295467 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:25.295469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:25.295478 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:13:25.295539 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:25.295540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:25.295552 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:25.295621 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:25.295767 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.296347 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.297309 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.297746 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.297828 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:25.297838 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:25.298021 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 11:13:25.298069 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 11:13:25.298365 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:25.295328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38171 (local address 127.0.98.1:39987)
0927 11:13:25.295437 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:25.295447 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:25.295487 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:25.295624 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:25.295625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:25.295627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:25.295694 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:25.295696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.296409 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.296410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:25.297232 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:25.297234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.297830 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.297831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:25.297890 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:25.297894 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:25.298246 (+ 352us) client_negotiation.cc:240] Negotiation successful
0927 11:13:25.298290 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":48,"threads_started":1}
I20260927 11:13:25.298336 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867a9f1351b6466a9606ea1d99c50ab8" candidate_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528b23b21f864e4dacae07daace39679" is_pre_election: true
I20260927 11:13:25.298496 7545 raft_consensus.cc:2454] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5b9fa623da4e6cb3389c7f86669d63 in term 0.
I20260927 11:13:25.298776 7343 leader_election.cc:302] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528b23b21f864e4dacae07daace39679, ff5b9fa623da4e6cb3389c7f86669d63; no voters:
I20260927 11:13:25.298887 7761 raft_consensus.cc:2781] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:25.298946 7761 raft_consensus.cc:482] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:25.298985 7761 raft_consensus.cc:3037] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:25.299081 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867a9f1351b6466a9606ea1d99c50ab8" candidate_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24208615a2e54f9d9638016054d89e17" is_pre_election: true
I20260927 11:13:25.299203 7681 raft_consensus.cc:2454] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5b9fa623da4e6cb3389c7f86669d63 in term 0.
I20260927 11:13:25.299717 7761 raft_consensus.cc:504] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } } peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } }
I20260927 11:13:25.299854 7761 leader_election.cc:288] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [CANDIDATE]: Term 1 election: Requested vote from peers 528b23b21f864e4dacae07daace39679 (127.0.98.2:34545), 24208615a2e54f9d9638016054d89e17 (127.0.98.3:38171)
I20260927 11:13:25.299995 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867a9f1351b6466a9606ea1d99c50ab8" candidate_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528b23b21f864e4dacae07daace39679"
I20260927 11:13:25.300029 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "867a9f1351b6466a9606ea1d99c50ab8" candidate_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24208615a2e54f9d9638016054d89e17"
I20260927 11:13:25.300074 7545 raft_consensus.cc:3037] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:25.300109 7681 raft_consensus.cc:3037] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:25.300725 7681 raft_consensus.cc:2454] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5b9fa623da4e6cb3389c7f86669d63 in term 1.
I20260927 11:13:25.300740 7545 raft_consensus.cc:2454] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5b9fa623da4e6cb3389c7f86669d63 in term 1.
I20260927 11:13:25.300879 7343 leader_election.cc:302] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 528b23b21f864e4dacae07daace39679, ff5b9fa623da4e6cb3389c7f86669d63; no voters:
I20260927 11:13:25.300979 7761 raft_consensus.cc:2781] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:25.301105 7761 raft_consensus.cc:686] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [term 1 LEADER]: Becoming Leader. State: Replica: ff5b9fa623da4e6cb3389c7f86669d63, State: Running, Role: LEADER
I20260927 11:13:25.301213 7761 consensus_queue.cc:237] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } } peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } }
I20260927 11:13:25.302000 6491 catalog_manager.cc:5495] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff5b9fa623da4e6cb3389c7f86669d63 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } health_report { overall_health: UNKNOWN } } }
I20260927 11:13:25.302567 7236 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: true } } }
I20260927 11:13:25.302676 7236 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:25.303301 7236 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df: attempting to promote NON_VOTER ca29eb8ceffc4dbb8dee1345574da3cb to VOTER
I20260927 11:13:25.303416 7236 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } }
I20260927 11:13:25.303838 7099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 FOLLOWER]: Refusing update from remote peer e3930554fc654a9eb440e352ed32c2df: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 11:13:25.303869 7285 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 1 LEARNER]: Refusing update from remote peer e3930554fc654a9eb440e352ed32c2df: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 11:13:25.304037 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260927 11:13:25.304122 7773 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260927 11:13:25.304658 7773 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ca29eb8ceffc4dbb8dee1345574da3cb (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } } }
I20260927 11:13:25.304909 7099 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ca29eb8ceffc4dbb8dee1345574da3cb (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } } }
I20260927 11:13:25.304970 7285 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ca29eb8ceffc4dbb8dee1345574da3cb (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } } }
I20260927 11:13:25.305589 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } } }
I20260927 11:13:25.305680 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:25.305743 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } } }
I20260927 11:13:25.305801 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:25.305851 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } } }
I20260927 11:13:25.305910 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:25.305909 7762 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } } }
I20260927 11:13:25.306001 7762 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:13:25.305972 7774 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e3930554fc654a9eb440e352ed32c2df" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } } }
I20260927 11:13:25.306080 7774 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:13:25.393920 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:25.390864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.0.98.1:36085)
0927 11:13:25.390919 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:25.390921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:25.390990 (+ 69us) server_negotiation.cc:407] Connection header received
0927 11:13:25.391092 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:25.391092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:25.391111 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:25.391168 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:25.391330 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.391893 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.393173 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.393352 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.393404 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:25.393606 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:25.393699 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:25.393757 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:25.393829 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 11:13:25.393845 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 11:13:25.395625 7681 raft_consensus.cc:1261] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17 [term 1 FOLLOWER]: Refusing update from remote peer ff5b9fa623da4e6cb3389c7f86669d63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:25.395658 7545 raft_consensus.cc:1261] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679 [term 1 FOLLOWER]: Refusing update from remote peer ff5b9fa623da4e6cb3389c7f86669d63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:25.396013 7769 consensus_queue.cc:1035] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:25.396149 7761 consensus_queue.cc:1035] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:25.398900 7776 mvcc.cc:205] Tried to move back new op lower bound from 7333919151698100224 to 7333919151314284544. Current Snapshot: MvccSnapshot[applied={T|T < 7333919151698100224}]
I20260927 11:13:25.400746 7778 mvcc.cc:205] Tried to move back new op lower bound from 7333919151698100224 to 7333919151314284544. Current Snapshot: MvccSnapshot[applied={T|T < 7333919151698100224}]
I20260927 11:13:25.401707 6490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:25.401808 6490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:25.404986 7777 mvcc.cc:205] Tried to move back new op lower bound from 7333919151698100224 to 7333919151314284544. Current Snapshot: MvccSnapshot[applied={T|T < 7333919151698100224}]
I20260927 11:13:25.405194 6490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3930554fc654a9eb440e352ed32c2df: Sending DeleteTablet for 3 replicas of tablet 867a9f1351b6466a9606ea1d99c50ab8
I20260927 11:13:25.405507 7388 tablet_service.cc:1514] Processing DeleteTablet for tablet 867a9f1351b6466a9606ea1d99c50ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54758
I20260927 11:13:25.405659 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet 867a9f1351b6466a9606ea1d99c50ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50306
I20260927 11:13:25.405718 7785 tablet_replica.cc:330] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63: stopping tablet replica
I20260927 11:13:25.405803 7785 raft_consensus.cc:2229] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:13:25.405895 7785 raft_consensus.cc:2258] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:25.405954 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet 867a9f1351b6466a9606ea1d99c50ab8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:25 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56068
I20260927 11:13:25.406333 7785 ts_tablet_manager.cc:1905] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:25.407235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 6459
I20260927 11:13:25.408103 7785 ts_tablet_manager.cc:1918] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:25.408164 7785 log.cc:1196] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-0/wal/wals/867a9f1351b6466a9606ea1d99c50ab8
I20260927 11:13:25.408414 7785 ts_tablet_manager.cc:1939] T 867a9f1351b6466a9606ea1d99c50ab8 P ff5b9fa623da4e6cb3389c7f86669d63: Deleting consensus metadata
I20260927 11:13:25.408874 7787 tablet_replica.cc:330] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17: stopping tablet replica
I20260927 11:13:25.408962 7787 raft_consensus.cc:2229] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:25.409066 7787 raft_consensus.cc:2258] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:25.409484 7787 ts_tablet_manager.cc:1905] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:25.411345 7787 ts_tablet_manager.cc:1918] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:25.411403 7787 log.cc:1196] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/867a9f1351b6466a9606ea1d99c50ab8
I20260927 11:13:25.411640 7787 ts_tablet_manager.cc:1939] T 867a9f1351b6466a9606ea1d99c50ab8 P 24208615a2e54f9d9638016054d89e17: Deleting consensus metadata
W20260927 11:13:25.414217 7343 connection.cc:535] server connection from 127.0.0.1:54758 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 11:13:25.414413 7786 tablet_replica.cc:330] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679: stopping tablet replica
I20260927 11:13:25.414537 7786 raft_consensus.cc:2229] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:25.414673 7786 raft_consensus.cc:2258] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:25.415098 7786 ts_tablet_manager.cc:1905] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:25.416749 7786 ts_tablet_manager.cc:1918] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:25.416872 7786 log.cc:1196] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/867a9f1351b6466a9606ea1d99c50ab8
I20260927 11:13:25.417184 7786 ts_tablet_manager.cc:1939] T 867a9f1351b6466a9606ea1d99c50ab8 P 528b23b21f864e4dacae07daace39679: Deleting consensus metadata
W20260927 11:13:25.417505 7479 connection.cc:441] server connection from 127.0.0.1:50306 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50306 (request call id 1) could send its response
I20260927 11:13:25.423813 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:25.406795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55814 (local address 127.0.98.60:41863)
0927 11:13:25.406833 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:25.406835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:25.414434 (+ 7599us) server_negotiation.cc:407] Connection header received
0927 11:13:25.414457 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:25.414458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:25.414479 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:25.414529 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:25.416844 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.417590 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.418472 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.418763 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.418796 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:25.420422 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:25.420423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:25.420424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:25.420430 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:25.422136 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:25.422488 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:25.422489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:25.422489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:25.422543 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:25.423033 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:25.423034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:25.423034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:25.423172 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:25.423265 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:25.423629 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 11:13:25.423698 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:13:25.427259 7140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:25.418302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.0.98.61:40661)
0927 11:13:25.418350 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:25.418352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:25.419241 (+ 889us) server_negotiation.cc:407] Connection header received
0927 11:13:25.419346 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:25.419347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:25.419358 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:25.419387 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:25.420200 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.420940 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.422034 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.422344 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.422386 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:25.424401 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:25.424402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:25.424402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:25.424409 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:25.426138 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:25.426488 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:25.426489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:25.426489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:25.426543 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:25.426733 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:25.426734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:25.426734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:25.426844 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:25.426956 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:25.427099 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 11:13:25.427163 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 11:13:25.431078 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:25.418890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.0.98.60:41863)
0927 11:13:25.418926 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:25.418928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:25.419802 (+ 874us) server_negotiation.cc:407] Connection header received
0927 11:13:25.422754 (+ 2952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:25.422755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:25.422767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:25.422797 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:25.424743 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.425518 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.426553 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:25.426881 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:25.426958 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:25.428423 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:25.428424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:25.428424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:25.428430 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:25.429993 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:25.430433 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:25.430434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:25.430434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:25.430486 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:25.430677 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:25.430678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:25.430678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:25.430763 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:25.430827 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:25.430955 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 11:13:25.431024 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:13:26.400425 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:26.400142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.2:50393)
0927 11:13:26.400274 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:26.400364 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
W20260927 11:13:26.400727 7592 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45085 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
I20260927 11:13:26.406227 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:26.406015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.1:38301)
0927 11:13:26.406141 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:26.406194 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
W20260927 11:13:26.406419 7455 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45085 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
I20260927 11:13:26.406635 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:26.406462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.3:41597)
0927 11:13:26.406547 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:26.406601 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":53,"threads_started":1}
W20260927 11:13:26.406831 7728 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45085 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
I20260927 11:13:26.904883 7799 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3930554fc654a9eb440e352ed32c2df)
I20260927 11:13:26.904985 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3930554fc654a9eb440e352ed32c2df)
I20260927 11:13:26.905009 7799 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } }
I20260927 11:13:26.905140 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } }
I20260927 11:13:26.905346 7799 leader_election.cc:288] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3930554fc654a9eb440e352ed32c2df (127.0.98.62:45085), ca29eb8ceffc4dbb8dee1345574da3cb (127.0.98.60:41863)
I20260927 11:13:26.905673 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:26.905461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:36820)
0927 11:13:26.905586 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:26.905628 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":44,"threads_started":1}
I20260927 11:13:26.905689 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:26.905426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:36804)
0927 11:13:26.905599 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:26.905646 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":49,"threads_started":1}
W20260927 11:13:26.905790 7076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
I20260927 11:13:26.905790 7800 leader_election.cc:288] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3930554fc654a9eb440e352ed32c2df (127.0.98.62:45085), bfc088fef7a4465e8a6176585e798392 (127.0.98.61:40661)
W20260927 11:13:26.906083 7262 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
I20260927 11:13:26.906215 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:26.906136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:36832)
0927 11:13:26.906171 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:26.906191 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 11:13:26.906303 7076 leader_election.cc:334] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3930554fc654a9eb440e352ed32c2df (127.0.98.62:45085): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
I20260927 11:13:26.907164 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:26.907058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:36834)
0927 11:13:26.907112 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:26.907141 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 11:13:26.907292 7262 leader_election.cc:334] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3930554fc654a9eb440e352ed32c2df (127.0.98.62:45085): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
I20260927 11:13:26.908756 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:26.905487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41863 (local address 127.0.0.1:55842)
0927 11:13:26.905865 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:26.905875 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:26.905944 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:26.906163 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:26.906164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:26.906166 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:26.906211 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:26.906213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:26.907040 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:26.907040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:26.907919 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:26.907921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:26.908566 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:26.908566 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:26.908607 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:26.908610 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:26.908649 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 11:13:26.908678 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":58,"threads_started":1}
I20260927 11:13:26.908759 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:26.905588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55842 (local address 127.0.98.60:41863)
0927 11:13:26.905668 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:26.905670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:26.905920 (+ 250us) server_negotiation.cc:407] Connection header received
0927 11:13:26.906020 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:26.906021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:26.906032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:26.906095 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:26.906294 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:26.906947 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:26.907977 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:26.908494 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:26.908564 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:26.908573 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:26.908654 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 11:13:26.908690 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65,"thread_start_us":41,"threads_started":1}
I20260927 11:13:26.908998 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:26.905725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.0.98.61:40661)
0927 11:13:26.905853 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:26.905855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:26.905863 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:13:26.905888 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:26.905889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:26.905900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:26.905975 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:26.906186 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:26.906803 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:26.907947 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:26.908494 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:26.908564 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:26.908574 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:26.908746 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 11:13:26.908782 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":48,"threads_started":1}
I20260927 11:13:26.909018 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfc088fef7a4465e8a6176585e798392" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" is_pre_election: true
I20260927 11:13:26.909109 7285 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfc088fef7a4465e8a6176585e798392 in term 1.
I20260927 11:13:26.909164 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:26.905629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40661 (local address 127.0.0.1:44768)
0927 11:13:26.905714 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:26.905723 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:26.905788 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:26.906025 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:26.906026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:26.906028 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:26.906105 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:26.906107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:26.906888 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:26.906889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:26.907853 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:26.907855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:26.908616 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:26.908617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:26.908673 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:26.908676 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:26.908729 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 11:13:26.908762 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":32,"threads_started":1}
I20260927 11:13:26.909314 7074 leader_election.cc:302] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bfc088fef7a4465e8a6176585e798392, ca29eb8ceffc4dbb8dee1345574da3cb; no voters: e3930554fc654a9eb440e352ed32c2df
I20260927 11:13:26.909327 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bfc088fef7a4465e8a6176585e798392" is_pre_election: true
I20260927 11:13:26.909410 7099 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca29eb8ceffc4dbb8dee1345574da3cb in term 1.
I20260927 11:13:26.909452 7799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 11:13:26.909498 7799 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e3930554fc654a9eb440e352ed32c2df)
I20260927 11:13:26.909526 7799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 1 FOLLOWER]: Advancing to term 2
I20260927 11:13:26.909529 7260 leader_election.cc:302] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bfc088fef7a4465e8a6176585e798392, ca29eb8ceffc4dbb8dee1345574da3cb; no voters: e3930554fc654a9eb440e352ed32c2df
I20260927 11:13:26.909632 7800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 11:13:26.909670 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 1 FOLLOWER]: Starting leader election (detected failure of leader e3930554fc654a9eb440e352ed32c2df)
I20260927 11:13:26.909693 7800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 1 FOLLOWER]: Advancing to term 2
I20260927 11:13:26.910313 7799 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } }
I20260927 11:13:26.910379 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } }
I20260927 11:13:26.910447 7799 leader_election.cc:288] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [CANDIDATE]: Term 2 election: Requested vote from peers e3930554fc654a9eb440e352ed32c2df (127.0.98.62:45085), ca29eb8ceffc4dbb8dee1345574da3cb (127.0.98.60:41863)
I20260927 11:13:26.910534 7800 leader_election.cc:288] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [CANDIDATE]: Term 2 election: Requested vote from peers e3930554fc654a9eb440e352ed32c2df (127.0.98.62:45085), bfc088fef7a4465e8a6176585e798392 (127.0.98.61:40661)
I20260927 11:13:26.910605 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfc088fef7a4465e8a6176585e798392" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb"
I20260927 11:13:26.910676 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:26.910597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:36856)
0927 11:13:26.910627 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:26.910651 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 11:13:26.910688 7285 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bfc088fef7a4465e8a6176585e798392 in current term 2: Already voted for candidate ca29eb8ceffc4dbb8dee1345574da3cb in this term.
I20260927 11:13:26.910809 7099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bfc088fef7a4465e8a6176585e798392"
I20260927 11:13:26.910912 7099 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ca29eb8ceffc4dbb8dee1345574da3cb in current term 2: Already voted for candidate bfc088fef7a4465e8a6176585e798392 in this term.
I20260927 11:13:26.910848 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:26.910507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:36848)
0927 11:13:26.910799 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:26.910827 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260927 11:13:26.911088 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:26.911011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:36868)
0927 11:13:26.911047 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:26.911069 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 11:13:26.911163 7262 leader_election.cc:334] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e3930554fc654a9eb440e352ed32c2df (127.0.98.62:45085): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
I20260927 11:13:26.911201 7262 leader_election.cc:302] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca29eb8ceffc4dbb8dee1345574da3cb; no voters: bfc088fef7a4465e8a6176585e798392, e3930554fc654a9eb440e352ed32c2df
I20260927 11:13:26.911201 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:26.911110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:36882)
0927 11:13:26.911156 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:26.911180 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 11:13:26.911257 7076 leader_election.cc:334] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e3930554fc654a9eb440e352ed32c2df (127.0.98.62:45085): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
I20260927 11:13:26.911295 7076 leader_election.cc:302] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bfc088fef7a4465e8a6176585e798392; no voters: ca29eb8ceffc4dbb8dee1345574da3cb, e3930554fc654a9eb440e352ed32c2df
I20260927 11:13:26.911288 7800 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 11:13:26.911353 7799 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 11:13:27.401381 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:27.401123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.2:45581)
0927 11:13:27.401287 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:27.401325 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260927 11:13:27.406855 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:27.406682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.1:51573)
0927 11:13:27.406793 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:27.406824 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260927 11:13:27.407250 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:27.407093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.3:56863)
0927 11:13:27.407183 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:27.407216 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":56,"threads_started":1}
I20260927 11:13:28.402170 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.401858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.2:36623)
0927 11:13:28.402039 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.402101 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":116,"threads_started":1}
I20260927 11:13:28.407431 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.407276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.1:43297)
0927 11:13:28.407368 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.407399 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":44,"threads_started":1}
I20260927 11:13:28.407786 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.407626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.3:45117)
0927 11:13:28.407726 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.407756 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
I20260927 11:13:28.486196 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:28.486315 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } }
I20260927 11:13:28.486542 7815 leader_election.cc:288] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3930554fc654a9eb440e352ed32c2df (127.0.98.62:45085), ca29eb8ceffc4dbb8dee1345574da3cb (127.0.98.60:41863)
I20260927 11:13:28.486742 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfc088fef7a4465e8a6176585e798392" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" is_pre_election: true
I20260927 11:13:28.486845 7285 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfc088fef7a4465e8a6176585e798392 in term 2.
I20260927 11:13:28.486896 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.486633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:37126)
0927 11:13:28.486791 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.486831 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260927 11:13:28.487015 7074 leader_election.cc:302] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfc088fef7a4465e8a6176585e798392, ca29eb8ceffc4dbb8dee1345574da3cb; no voters:
I20260927 11:13:28.487090 7815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 11:13:28.487138 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:28.487186 7815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 2 FOLLOWER]: Advancing to term 3
I20260927 11:13:28.487278 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.487189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:37138)
0927 11:13:28.487230 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.487252 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 11:13:28.487385 7076 leader_election.cc:334] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3930554fc654a9eb440e352ed32c2df (127.0.98.62:45085): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
I20260927 11:13:28.487963 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } }
I20260927 11:13:28.488089 7815 leader_election.cc:288] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [CANDIDATE]: Term 3 election: Requested vote from peers e3930554fc654a9eb440e352ed32c2df (127.0.98.62:45085), ca29eb8ceffc4dbb8dee1345574da3cb (127.0.98.60:41863)
I20260927 11:13:28.488246 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.488164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:37152)
0927 11:13:28.488195 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.488224 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 11:13:28.488233 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bfc088fef7a4465e8a6176585e798392" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb"
I20260927 11:13:28.488317 7285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 2 FOLLOWER]: Advancing to term 3
I20260927 11:13:28.488538 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.488448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:37154)
0927 11:13:28.488499 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.488520 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 11:13:28.488642 7076 leader_election.cc:334] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e3930554fc654a9eb440e352ed32c2df (127.0.98.62:45085): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
I20260927 11:13:28.488986 7285 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfc088fef7a4465e8a6176585e798392 in term 3.
I20260927 11:13:28.489166 7074 leader_election.cc:302] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bfc088fef7a4465e8a6176585e798392, ca29eb8ceffc4dbb8dee1345574da3cb; no voters: e3930554fc654a9eb440e352ed32c2df
I20260927 11:13:28.489277 7815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 11:13:28.489384 7815 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [term 3 LEADER]: Becoming Leader. State: Replica: bfc088fef7a4465e8a6176585e798392, State: Running, Role: LEADER
I20260927 11:13:28.489456 7815 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } }
I20260927 11:13:28.489791 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfc088fef7a4465e8a6176585e798392. Latest consensus state: current_term: 3 leader_uuid: "bfc088fef7a4465e8a6176585e798392" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } } }
I20260927 11:13:28.489880 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:28.490099 7820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:28.490516 7820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:28.490572 7820 catalog_manager.cc:1259] Loaded cluster ID: b56755796adc430a96201df52bbd5158
I20260927 11:13:28.490621 7820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:13:28.490809 7820 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:28.490893 7820 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: Loaded TSK: 0
I20260927 11:13:28.491039 7820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:13:28.505867 7088 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:13:28.507354 7088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:28.508162 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.508070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:37164)
0927 11:13:28.508113 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.508142 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 11:13:28.508178 7285 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [term 3 FOLLOWER]: Refusing update from remote peer bfc088fef7a4465e8a6176585e798392: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 11:13:28.508461 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.508373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.0.1:37170)
0927 11:13:28.508419 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.508445 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 11:13:28.508424 7818 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 11:13:28.508544 7076 consensus_peers.cc:487] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 -> Peer e3930554fc654a9eb440e352ed32c2df (127.0.98.62:45085): Couldn't send request to peer e3930554fc654a9eb440e352ed32c2df. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 11:13:28.509003 7822 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfc088fef7a4465e8a6176585e798392. Latest consensus state: current_term: 3 leader_uuid: "bfc088fef7a4465e8a6176585e798392" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } } }
I20260927 11:13:28.509100 7822 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:13:28.509514 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bfc088fef7a4465e8a6176585e798392" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } } }
I20260927 11:13:28.509608 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:28.509735 7822 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bfc088fef7a4465e8a6176585e798392" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "e3930554fc654a9eb440e352ed32c2df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45085 } } peers { permanent_uuid: "bfc088fef7a4465e8a6176585e798392" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40661 } attrs { promote: false } } peers { permanent_uuid: "ca29eb8ceffc4dbb8dee1345574da3cb" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41863 } attrs { promote: false } } }
I20260927 11:13:28.509819 7822 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca29eb8ceffc4dbb8dee1345574da3cb [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:13:28.514299 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.511238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36085 (local address 127.0.0.1:54770)
0927 11:13:28.511271 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.511281 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:28.511328 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:28.511709 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:28.511710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:28.511711 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:28.511756 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:28.511757 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.512544 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.512545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:28.513296 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:28.513298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.514046 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.514047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:28.514123 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:28.514127 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:28.514203 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 11:13:28.514240 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260927 11:13:28.514323 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.511343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54770 (local address 127.0.98.1:36085)
0927 11:13:28.511439 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:28.511441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:28.511450 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:13:28.511475 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:28.511476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:28.511489 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:28.511844 (+ 355us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:28.511881 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.512474 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.513391 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.513967 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.514035 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:28.514066 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:28.514211 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 11:13:28.514255 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":52,"threads_started":1}
I20260927 11:13:28.514627 7388 tablet_service.cc:1467] Processing CreateTablet for tablet 921dd55155bb40b29aca7b38be68c0d1 (DEFAULT_TABLE table=test-table [id=042a2f5dd8d94f9c8d5e4d922f53fb1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:28.514679 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.511295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.0.98.2:34545)
0927 11:13:28.511399 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:28.511402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:28.511529 (+ 127us) server_negotiation.cc:407] Connection header received
0927 11:13:28.511624 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:28.511625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:28.511636 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:28.511674 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:28.511858 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.512372 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.513717 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.514325 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.514391 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:28.514403 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:28.514577 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 11:13:28.514618 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":60,"threads_started":1}
I20260927 11:13:28.514693 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.511239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34545 (local address 127.0.0.1:50320)
0927 11:13:28.511474 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.511484 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:28.511551 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:28.511748 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:28.511748 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:28.511750 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:28.511795 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:28.511796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.512439 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.512440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:28.513611 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:28.513613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.514455 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.514456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:28.514518 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:28.514522 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:28.514574 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 11:13:28.514617 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":135,"thread_start_us":60,"threads_started":1}
I20260927 11:13:28.514740 7388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921dd55155bb40b29aca7b38be68c0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:28.514905 7525 tablet_service.cc:1467] Processing CreateTablet for tablet 921dd55155bb40b29aca7b38be68c0d1 (DEFAULT_TABLE table=test-table [id=042a2f5dd8d94f9c8d5e4d922f53fb1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:28.515048 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921dd55155bb40b29aca7b38be68c0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:28.515739 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.511540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38171 (local address 127.0.0.1:56074)
0927 11:13:28.512584 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.512593 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:28.512632 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:28.512855 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:28.512856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:28.512858 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:28.512890 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:28.512891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.513776 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.513777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:28.514776 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:28.514778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.515507 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.515507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:28.515582 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:28.515586 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:28.515639 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 11:13:28.515671 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"thread_start_us":57,"threads_started":1}
I20260927 11:13:28.515755 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.511597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.0.98.3:38171)
0927 11:13:28.512716 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:28.512718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:28.512725 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:13:28.512747 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:28.512748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:28.512757 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:28.512817 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:28.512955 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.513710 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.514885 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.515437 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.515505 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:28.515514 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:28.515644 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 11:13:28.515684 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1100,"thread_start_us":51,"threads_started":1}
I20260927 11:13:28.515934 7661 tablet_service.cc:1467] Processing CreateTablet for tablet 921dd55155bb40b29aca7b38be68c0d1 (DEFAULT_TABLE table=test-table [id=042a2f5dd8d94f9c8d5e4d922f53fb1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:28.516034 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 921dd55155bb40b29aca7b38be68c0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:28.516084 7833 tablet_bootstrap.cc:492] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63: Bootstrap starting.
I20260927 11:13:28.516570 7833 tablet_bootstrap.cc:654] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:28.516834 7834 tablet_bootstrap.cc:492] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679: Bootstrap starting.
I20260927 11:13:28.517113 7834 tablet_bootstrap.cc:654] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:28.517514 7837 tablet_bootstrap.cc:492] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17: Bootstrap starting.
I20260927 11:13:28.517750 7833 tablet_bootstrap.cc:492] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63: No bootstrap required, opened a new log
I20260927 11:13:28.517803 7833 ts_tablet_manager.cc:1397] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:28.517902 7837 tablet_bootstrap.cc:654] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:28.517921 7833 raft_consensus.cc:348] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } } peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } }
I20260927 11:13:28.517992 7833 raft_consensus.cc:374] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:28.518024 7833 raft_consensus.cc:729] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5b9fa623da4e6cb3389c7f86669d63, State: Initialized, Role: FOLLOWER
I20260927 11:13:28.518095 7833 consensus_queue.cc:260] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } } peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } }
I20260927 11:13:28.518303 7833 ts_tablet_manager.cc:1428] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:28.518424 7837 tablet_bootstrap.cc:492] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17: No bootstrap required, opened a new log
I20260927 11:13:28.518469 7837 ts_tablet_manager.cc:1397] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:28.518549 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.518460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.1:41497)
0927 11:13:28.518499 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.518522 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 11:13:28.518571 7837 raft_consensus.cc:348] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } } peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } }
I20260927 11:13:28.518651 7837 raft_consensus.cc:374] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:28.518666 7834 tablet_bootstrap.cc:492] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679: No bootstrap required, opened a new log
I20260927 11:13:28.518672 7837 raft_consensus.cc:729] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24208615a2e54f9d9638016054d89e17, State: Initialized, Role: FOLLOWER
I20260927 11:13:28.518704 7834 ts_tablet_manager.cc:1397] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:28.518712 7837 consensus_queue.cc:260] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } } peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } }
I20260927 11:13:28.518872 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.518776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.1:39481)
0927 11:13:28.518824 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.518845 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 11:13:28.518877 7837 ts_tablet_manager.cc:1428] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:28.518865 7834 raft_consensus.cc:348] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } } peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } }
I20260927 11:13:28.518937 7834 raft_consensus.cc:374] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:28.518956 7834 raft_consensus.cc:729] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 528b23b21f864e4dacae07daace39679, State: Initialized, Role: FOLLOWER
I20260927 11:13:28.518999 7834 consensus_queue.cc:260] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } } peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } }
I20260927 11:13:28.519209 7834 ts_tablet_manager.cc:1428] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:28.519448 7456 heartbeater.cc:503] Master 127.0.98.61:40661 was elected leader, sending a full tablet report...
I20260927 11:13:28.519482 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.519396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.2:49041)
0927 11:13:28.519430 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.519459 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260927 11:13:28.519745 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.519641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.2:60193)
0927 11:13:28.519675 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.519706 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 11:13:28.520095 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.519800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.3:60273)
0927 11:13:28.520036 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.520065 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214}
I20260927 11:13:28.520179 7729 heartbeater.cc:503] Master 127.0.98.61:40661 was elected leader, sending a full tablet report...
I20260927 11:13:28.713980 7840 raft_consensus.cc:482] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:28.714115 7840 raft_consensus.cc:504] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } } peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } }
I20260927 11:13:28.714448 7840 leader_election.cc:288] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 528b23b21f864e4dacae07daace39679 (127.0.98.2:34545), ff5b9fa623da4e6cb3389c7f86669d63 (127.0.98.1:36085)
I20260927 11:13:28.718382 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.714722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37909 (local address 127.0.98.2:34545)
0927 11:13:28.714833 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:28.714835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:28.714845 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:13:28.714911 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:28.714912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:28.714923 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:28.714993 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:28.715201 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.715948 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.717462 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.718014 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.718118 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:28.718130 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:28.718275 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 11:13:28.718313 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 11:13:28.718382 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.714730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44221 (local address 127.0.98.1:36085)
0927 11:13:28.714778 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:28.714780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:28.714959 (+ 179us) server_negotiation.cc:407] Connection header received
0927 11:13:28.715015 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:28.715034 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:28.715053 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:28.715095 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:28.715223 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.715968 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.717463 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.718014 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.718118 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:28.718130 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:28.718276 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 11:13:28.718313 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:13:28.718405 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.714633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36085 (local address 127.0.98.3:44221)
0927 11:13:28.714883 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.714892 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:28.714949 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:28.715106 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:28.715106 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:28.715108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:28.715156 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:28.715158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.716043 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.716043 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:28.717369 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:28.717372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.718116 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.718117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:28.718195 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:28.718199 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:28.718255 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 11:13:28.718304 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":105,"thread_start_us":137,"threads_started":1}
I20260927 11:13:28.718408 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.714633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34545 (local address 127.0.98.3:37909)
0927 11:13:28.714727 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.714742 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:28.714837 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:28.715006 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:28.715007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:28.715009 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:28.715100 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:28.715101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.716039 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.716040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:28.717369 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:28.717371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.718118 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.718118 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:28.718196 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:28.718198 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:13:28.718255 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 11:13:28.718304 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":76}
I20260927 11:13:28.718653 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921dd55155bb40b29aca7b38be68c0d1" candidate_uuid: "24208615a2e54f9d9638016054d89e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" is_pre_election: true
I20260927 11:13:28.718803 7408 raft_consensus.cc:2454] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24208615a2e54f9d9638016054d89e17 in term 0.
I20260927 11:13:28.718847 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921dd55155bb40b29aca7b38be68c0d1" candidate_uuid: "24208615a2e54f9d9638016054d89e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528b23b21f864e4dacae07daace39679" is_pre_election: true
I20260927 11:13:28.718947 7545 raft_consensus.cc:2454] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24208615a2e54f9d9638016054d89e17 in term 0.
I20260927 11:13:28.718991 7617 leader_election.cc:302] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24208615a2e54f9d9638016054d89e17, ff5b9fa623da4e6cb3389c7f86669d63; no voters:
I20260927 11:13:28.719107 7840 raft_consensus.cc:2781] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:28.719141 7840 raft_consensus.cc:482] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:28.719178 7840 raft_consensus.cc:3037] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:28.719826 7840 raft_consensus.cc:504] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } } peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } }
I20260927 11:13:28.719959 7840 leader_election.cc:288] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [CANDIDATE]: Term 1 election: Requested vote from peers 528b23b21f864e4dacae07daace39679 (127.0.98.2:34545), ff5b9fa623da4e6cb3389c7f86669d63 (127.0.98.1:36085)
I20260927 11:13:28.720088 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921dd55155bb40b29aca7b38be68c0d1" candidate_uuid: "24208615a2e54f9d9638016054d89e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "528b23b21f864e4dacae07daace39679"
I20260927 11:13:28.720106 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "921dd55155bb40b29aca7b38be68c0d1" candidate_uuid: "24208615a2e54f9d9638016054d89e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5b9fa623da4e6cb3389c7f86669d63"
I20260927 11:13:28.720170 7545 raft_consensus.cc:3037] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:28.720193 7408 raft_consensus.cc:3037] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:28.720798 7408 raft_consensus.cc:2454] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24208615a2e54f9d9638016054d89e17 in term 1.
I20260927 11:13:28.720813 7545 raft_consensus.cc:2454] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24208615a2e54f9d9638016054d89e17 in term 1.
I20260927 11:13:28.720944 7617 leader_election.cc:302] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24208615a2e54f9d9638016054d89e17, ff5b9fa623da4e6cb3389c7f86669d63; no voters:
I20260927 11:13:28.721055 7840 raft_consensus.cc:2781] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:28.721190 7840 raft_consensus.cc:686] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [term 1 LEADER]: Becoming Leader. State: Replica: 24208615a2e54f9d9638016054d89e17, State: Running, Role: LEADER
I20260927 11:13:28.721273 7840 consensus_queue.cc:237] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } } peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } }
I20260927 11:13:28.721843 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.721752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.3:51415)
0927 11:13:28.721791 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.721815 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 11:13:28.722214 7089 catalog_manager.cc:5495] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24208615a2e54f9d9638016054d89e17 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "24208615a2e54f9d9638016054d89e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24208615a2e54f9d9638016054d89e17" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 } health_report { overall_health: UNKNOWN } } }
I20260927 11:13:28.739094 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.737119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56090 (local address 127.0.98.3:38171)
0927 11:13:28.737165 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:28.737167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:28.737173 (+ 6us) server_negotiation.cc:407] Connection header received
0927 11:13:28.737199 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:28.737200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:28.737216 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:28.737272 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:28.737377 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.737822 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.738520 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:28.738693 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:28.738743 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:28.738871 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:28.738955 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:28.739004 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:28.739038 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 11:13:28.739052 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 11:13:28.740041 7545 raft_consensus.cc:1261] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679 [term 1 FOLLOWER]: Refusing update from remote peer 24208615a2e54f9d9638016054d89e17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:28.740137 7408 raft_consensus.cc:1261] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63 [term 1 FOLLOWER]: Refusing update from remote peer 24208615a2e54f9d9638016054d89e17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:28.740257 7840 consensus_queue.cc:1035] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "528b23b21f864e4dacae07daace39679" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:28.740413 7843 consensus_queue.cc:1035] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5b9fa623da4e6cb3389c7f86669d63" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:28.740751 7593 heartbeater.cc:503] Master 127.0.98.61:40661 was elected leader, sending a full tablet report...
I20260927 11:13:28.741328 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.741103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.2:33863)
0927 11:13:28.741273 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.741304 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260927 11:13:28.741564 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.741477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.1:38679)
0927 11:13:28.741516 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.741538 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 11:13:28.742333 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.742227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.2:57295)
0927 11:13:28.742263 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.742303 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 11:13:28.742585 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.742500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.3:55573)
0927 11:13:28.742534 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.742557 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 11:13:28.744160 7089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:28.744253 7089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:28.744536 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:28.744460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45085 (local address 127.0.98.1:54569)
0927 11:13:28.744490 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.744512 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 11:13:28.745500 7089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bfc088fef7a4465e8a6176585e798392: Sending DeleteTablet for 3 replicas of tablet 921dd55155bb40b29aca7b38be68c0d1
I20260927 11:13:28.745662 7388 tablet_service.cc:1514] Processing DeleteTablet for tablet 921dd55155bb40b29aca7b38be68c0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:28 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54770
I20260927 11:13:28.745818 7387 tablet_service.cc:1514] Processing DeleteTablet for tablet 921dd55155bb40b29aca7b38be68c0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:28 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54770
I20260927 11:13:28.745908 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet 921dd55155bb40b29aca7b38be68c0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:28 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:56074
I20260927 11:13:28.745947 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet 921dd55155bb40b29aca7b38be68c0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:28 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:50320
I20260927 11:13:28.746110 7857 tablet_replica.cc:330] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17: stopping tablet replica
I20260927 11:13:28.746130 7856 tablet_replica.cc:330] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63: stopping tablet replica
I20260927 11:13:28.746181 7856 raft_consensus.cc:2229] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:28.746228 7856 raft_consensus.cc:2258] T 921dd55155bb40b29aca7b38be68c0d1 P ff5b9fa623da4e6cb3389c7f86669d63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:28.746294 7857 raft_consensus.cc:2229] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:13:28.746417 7857 raft_consensus.cc:2258] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:28.746470 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 7237
I20260927 11:13:28.747269 7857 ts_tablet_manager.cc:1905] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:28.748235 7858 tablet_replica.cc:330] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679: stopping tablet replica
I20260927 11:13:28.748297 7858 raft_consensus.cc:2229] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:28.748343 7858 raft_consensus.cc:2258] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:28.748625 7858 ts_tablet_manager.cc:1905] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:28.749025 7857 ts_tablet_manager.cc:1918] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:28.749127 7857 log.cc:1196] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/921dd55155bb40b29aca7b38be68c0d1
I20260927 11:13:28.749410 7857 ts_tablet_manager.cc:1939] T 921dd55155bb40b29aca7b38be68c0d1 P 24208615a2e54f9d9638016054d89e17: Deleting consensus metadata
I20260927 11:13:28.749748 7075 catalog_manager.cc:4841] TS 24208615a2e54f9d9638016054d89e17 (127.0.98.3:38171): tablet 921dd55155bb40b29aca7b38be68c0d1 (table test-table [id=042a2f5dd8d94f9c8d5e4d922f53fb1c]) successfully deleted
I20260927 11:13:28.750412 7858 ts_tablet_manager.cc:1918] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:28.750473 7858 log.cc:1196] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/921dd55155bb40b29aca7b38be68c0d1
I20260927 11:13:28.750701 7858 ts_tablet_manager.cc:1939] T 921dd55155bb40b29aca7b38be68c0d1 P 528b23b21f864e4dacae07daace39679: Deleting consensus metadata
I20260927 11:13:28.751549 7075 catalog_manager.cc:4841] TS 528b23b21f864e4dacae07daace39679 (127.0.98.2:34545): tablet 921dd55155bb40b29aca7b38be68c0d1 (table test-table [id=042a2f5dd8d94f9c8d5e4d922f53fb1c]) successfully deleted
W20260927 11:13:28.753710 7831 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:28.753600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36085 (local address 127.0.0.1:54780)
0927 11:13:28.753638 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:28.753662 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36085: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 11:13:28.753824 7076 catalog_manager.cc:4584] TS ff5b9fa623da4e6cb3389c7f86669d63 (127.0.98.1:36085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 921dd55155bb40b29aca7b38be68c0d1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36085: connect: Connection reset by peer (error 104)
I20260927 11:13:28.753965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 7462
I20260927 11:13:28.760025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 7598
I20260927 11:13:28.765473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 6955
I20260927 11:13:28.769901 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 7143
2026-09-27T11:13:28Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6850 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:13:28 dist-test-slave-9hq9 krb5kdc[7861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:13:28 dist-test-slave-9hq9 krb5kdc[7861](info): set up 2 sockets
Sep 27 11:13:28 dist-test-slave-9hq9 krb5kdc[7861](info): commencing operation
krb5kdc: starting...
W20260927 11:13:29.786547 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:13:29 dist-test-slave-9hq9 krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:13:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:29Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 11:13:29.896481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34979
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:29.973376 7874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:29.973599 7874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:29.973634 7874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:29.974794 7874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:29.974860 7874 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 11:13:29.974882 7874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:29.974905 7874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:29.974934 7874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:29.974958 7874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:29.976372 7874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34979
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:29.976642 7874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:29.976871 7874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:29.981268 7879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:29.981380 7880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:29.981323 7882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:29.981395 7874 server_base.cc:1034] running on GCE node
I20260927 11:13:29.981745 7874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:29.981985 7874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:29.983187 7874 hybrid_clock.cc:648] HybridClock initialized: now 1790507609983168 us; error 36 us; skew 500 ppm
Sep 27 11:13:29 dist-test-slave-9hq9 krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507609, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:29.985571 7874 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 11:13:29.986006 7874 webserver.cc:460] Webserver started at http://127.0.98.62:42891/ using document root <none> and password file <none>
I20260927 11:13:29.986271 7874 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:29.986338 7874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:29.986480 7874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:29.987423 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "1adde35a5d7e42e39e4231f247b87f42"
format_stamp: "Formatted at 2026-09-27 11:13:29 on dist-test-slave-9hq9"
server_key: "63ec4c07813abc1e4fb7549d00b17ccf"
server_key_iv: "2be6ac6e1d59bcf11ef399e374dd01d6"
server_key_version: "encryptionkey@0"
I20260927 11:13:29.987795 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "1adde35a5d7e42e39e4231f247b87f42"
format_stamp: "Formatted at 2026-09-27 11:13:29 on dist-test-slave-9hq9"
server_key: "63ec4c07813abc1e4fb7549d00b17ccf"
server_key_iv: "2be6ac6e1d59bcf11ef399e374dd01d6"
server_key_version: "encryptionkey@0"
I20260927 11:13:29.989456 7874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:13:29.990268 7889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:29.990453 7874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:29.990548 7874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "1adde35a5d7e42e39e4231f247b87f42"
format_stamp: "Formatted at 2026-09-27 11:13:29 on dist-test-slave-9hq9"
server_key: "63ec4c07813abc1e4fb7549d00b17ccf"
server_key_iv: "2be6ac6e1d59bcf11ef399e374dd01d6"
server_key_version: "encryptionkey@0"
I20260927 11:13:29.990643 7874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:30.002271 7874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:30.004415 7874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:30.004570 7874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:30.009140 7874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34979
I20260927 11:13:30.009186 7951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34979 every 8 connection(s)
I20260927 11:13:30.009459 7874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:30.009980 7952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:30.010684 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 7874
I20260927 11:13:30.010759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:30.010929 392 external_mini_cluster.cc:1428] Setting key 49c6662dab109634659d7eb72a9b56e5
I20260927 11:13:30.012682 7952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42: Bootstrap starting.
I20260927 11:13:30.013228 7952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:30.013506 7952 log.cc:824] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:30.014196 7952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42: No bootstrap required, opened a new log
I20260927 11:13:30.015903 7952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adde35a5d7e42e39e4231f247b87f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } }
I20260927 11:13:30.016048 7952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:30.016103 7952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1adde35a5d7e42e39e4231f247b87f42, State: Initialized, Role: FOLLOWER
I20260927 11:13:30.016242 7952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adde35a5d7e42e39e4231f247b87f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } }
I20260927 11:13:30.016357 7952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:30.016402 7952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:30.016448 7952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:30.017067 7952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adde35a5d7e42e39e4231f247b87f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } }
I20260927 11:13:30.017184 7952 leader_election.cc:302] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1adde35a5d7e42e39e4231f247b87f42; no voters:
I20260927 11:13:30.017333 7952 leader_election.cc:288] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [CANDIDATE]: Term 1 election: Requested vote from peers
Sep 27 11:13:30 dist-test-slave-9hq9 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:13:30.017386 7957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:30.017586 7952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:30.017617 7957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [term 1 LEADER]: Becoming Leader. State: Replica: 1adde35a5d7e42e39e4231f247b87f42, State: Running, Role: LEADER
I20260927 11:13:30.017809 7957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adde35a5d7e42e39e4231f247b87f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } }
I20260927 11:13:30.018188 7958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1adde35a5d7e42e39e4231f247b87f42. Latest consensus state: current_term: 1 leader_uuid: "1adde35a5d7e42e39e4231f247b87f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adde35a5d7e42e39e4231f247b87f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } }
I20260927 11:13:30.018339 7958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:30.018188 7959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1adde35a5d7e42e39e4231f247b87f42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1adde35a5d7e42e39e4231f247b87f42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34979 } } }
I20260927 11:13:30.018538 7959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42 [sys.catalog]: This master's current role is: LEADER
W20260927 11:13:30.019835 7971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:13:30.019899 7971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:13:30.021656 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.011417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.0.98.62:34979)
0927 11:13:30.011598 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:30.011608 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:30.011638 (+ 30us) server_negotiation.cc:407] Connection header received
0927 11:13:30.012034 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:30.012039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:30.012081 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:30.012222 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:30.012899 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.013866 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.014718 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.015140 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.015270 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.017965 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:30.017969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:30.017970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:30.017981 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:30.020212 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.020583 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.020585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.020585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.020661 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.020859 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.020859 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.020860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.021065 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:30.021163 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:30.021304 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 11:13:30.021394 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260927 11:13:30.026181 7972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:30.026382 7972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:30.027792 7972 catalog_manager.cc:1347] Generated new cluster ID: 08cc7c645ca641689a7a9cd3e8f81be0
I20260927 11:13:30.027839 7972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:13:30.033565 7972 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:30.034232 7972 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:30.042465 7972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42: Generated new TSK 0
I20260927 11:13:30.042618 7972 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 11:13:30.070541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34979
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:30.147750 7978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:30.147965 7978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:30.148001 7978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:30.149132 7978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:30.149195 7978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:30.149231 7978 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:13:30.149284 7978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:30.150796 7978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:30.151031 7978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:30.151228 7978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:13:30.155953 7978 server_base.cc:1034] running on GCE node
W20260927 11:13:30.156000 7984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:30.155939 7983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:30.156055 7986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:30.156340 7978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:30.156560 7978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:30.157716 7978 hybrid_clock.cc:648] HybridClock initialized: now 1790507610157704 us; error 27 us; skew 500 ppm
Sep 27 11:13:30 dist-test-slave-9hq9 krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:30.160151 7978 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 11:13:30.160588 7978 webserver.cc:460] Webserver started at http://127.0.98.1:45811/ using document root <none> and password file <none>
I20260927 11:13:30.160801 7978 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:30.160867 7978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:30.161018 7978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:30.161968 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "8210201d68e843cc90b1d2e08323e5c0"
format_stamp: "Formatted at 2026-09-27 11:13:30 on dist-test-slave-9hq9"
server_key: "0eeb78e8cbd6e44226e12846a080d3d1"
server_key_iv: "786420d40da245a65535b5f7da0a3daa"
server_key_version: "encryptionkey@0"
I20260927 11:13:30.162400 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8210201d68e843cc90b1d2e08323e5c0"
format_stamp: "Formatted at 2026-09-27 11:13:30 on dist-test-slave-9hq9"
server_key: "0eeb78e8cbd6e44226e12846a080d3d1"
server_key_iv: "786420d40da245a65535b5f7da0a3daa"
server_key_version: "encryptionkey@0"
I20260927 11:13:30.164050 7978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:30.165162 7993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:30.165344 7978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:30.165411 7978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "8210201d68e843cc90b1d2e08323e5c0"
format_stamp: "Formatted at 2026-09-27 11:13:30 on dist-test-slave-9hq9"
server_key: "0eeb78e8cbd6e44226e12846a080d3d1"
server_key_iv: "786420d40da245a65535b5f7da0a3daa"
server_key_version: "encryptionkey@0"
I20260927 11:13:30.165460 7978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:30.175554 7978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:30.177634 7978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:30.177726 7978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:30.178265 7978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:30.178309 7978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:30.178337 7978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:30.178357 7978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 11:13:30 dist-test-slave-9hq9 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:13:30.185436 7978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41965
I20260927 11:13:30.185737 7978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:13:30.185750 8108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41965 every 8 connection(s)
I20260927 11:13:30.187229 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.178579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60371 (local address 127.0.98.62:34979)
0927 11:13:30.178626 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:30.178629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:30.178888 (+ 259us) server_negotiation.cc:407] Connection header received
0927 11:13:30.179513 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:30.179514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:30.179523 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:30.179568 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:30.180377 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.180848 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.181798 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.182004 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.182123 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.184619 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:30.184620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:30.184620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:30.184628 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:30.186301 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.186625 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.186626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.186627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.186669 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.186820 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.186820 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.186820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.186904 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:30.186945 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:30.187120 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 11:13:30.187175 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:13:30.187379 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.178583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.1:60371)
0927 11:13:30.178744 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:30.178788 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:30.179415 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:30.179640 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:30.179643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:30.180001 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:30.180280 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.180285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.180956 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.180958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.181698 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.181700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.182133 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.182134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.182246 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.182703 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:30.182713 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:30.184486 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:30.186354 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:30.186357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:30.186358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:30.186551 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:30.186706 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:30.186706 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:30.186707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:30.186777 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:30.186938 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:30.186940 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:30.187032 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 11:13:30.187097 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 11:13:30.187192 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260927 11:13:30.187808 8109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34979
I20260927 11:13:30.187891 8109 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:30.188045 8109 heartbeater.cc:511] Master 127.0.98.62:34979 requested a full tablet report, sending...
I20260927 11:13:30.188622 7905 ts_manager.cc:194] Registered new tserver with Master: 8210201d68e843cc90b1d2e08323e5c0 (127.0.98.1:41965)
I20260927 11:13:30.189239 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60371
I20260927 11:13:30.194280 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.188761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60175 (local address 127.0.98.62:34979)
0927 11:13:30.188793 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:30.188795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:30.189525 (+ 730us) server_negotiation.cc:407] Connection header received
0927 11:13:30.189602 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:30.189602 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:30.189613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:30.189661 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:30.189936 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.190490 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.191137 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.191298 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.191346 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.192178 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:30.192178 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:30.192179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:30.192183 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:30.193309 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.193623 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.193624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.193624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.193682 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.193841 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.193841 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.193841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.193943 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:30.194007 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:30.194182 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 11:13:30.194231 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:13:30.194289 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.188705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.1:60175)
0927 11:13:30.189449 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:30.189460 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:30.189522 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:30.189684 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:30.189684 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:30.189824 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:30.189869 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.189870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.190599 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.190599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.191076 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.191078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.191358 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.191358 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.191418 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.191842 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:30.191849 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:30.192115 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:30.193374 (+ 1259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:30.193375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:30.193375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:30.193520 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:30.193738 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:30.193739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:30.193739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:30.193787 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:30.194002 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:30.194003 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:30.194089 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 11:13:30.194150 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 11:13:30.194227 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":74,"threads_started":1}
I20260927 11:13:30.194834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 7978
I20260927 11:13:30.194960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:13:30.195148 392 external_mini_cluster.cc:1428] Setting key 24c152c2e1fcce680ccb026c8aaaf9fb
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 11:13:30.217190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34979
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:30.294766 8118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:30.294965 8118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:30.294998 8118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:30.296025 8118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:30.296085 8118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:30.296115 8118 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:13:30.296151 8118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:30.297518 8118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:30.297765 8118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:30.297978 8118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:30.302384 8126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:30.302412 8124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:30.302474 8118 server_base.cc:1034] running on GCE node
W20260927 11:13:30.302434 8123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:30.302737 8118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:30.302969 8118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:30.304158 8118 hybrid_clock.cc:648] HybridClock initialized: now 1790507610304156 us; error 33 us; skew 500 ppm
Sep 27 11:13:30 dist-test-slave-9hq9 krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:30.306593 8118 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 11:13:30.307016 8118 webserver.cc:460] Webserver started at http://127.0.98.2:37447/ using document root <none> and password file <none>
I20260927 11:13:30.307224 8118 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:30.307284 8118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:30.307406 8118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:30.308327 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "d8f97afedb2f47858a02dcd706afcb75"
format_stamp: "Formatted at 2026-09-27 11:13:30 on dist-test-slave-9hq9"
server_key: "646a689a90e9bd24e050d1dfc71c1ec6"
server_key_iv: "a02f996d37ee39790a25f8992989867a"
server_key_version: "encryptionkey@0"
I20260927 11:13:30.308665 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d8f97afedb2f47858a02dcd706afcb75"
format_stamp: "Formatted at 2026-09-27 11:13:30 on dist-test-slave-9hq9"
server_key: "646a689a90e9bd24e050d1dfc71c1ec6"
server_key_iv: "a02f996d37ee39790a25f8992989867a"
server_key_version: "encryptionkey@0"
I20260927 11:13:30.309903 8118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:13:30.310657 8133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:30.310823 8118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:30.310911 8118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "d8f97afedb2f47858a02dcd706afcb75"
format_stamp: "Formatted at 2026-09-27 11:13:30 on dist-test-slave-9hq9"
server_key: "646a689a90e9bd24e050d1dfc71c1ec6"
server_key_iv: "a02f996d37ee39790a25f8992989867a"
server_key_version: "encryptionkey@0"
I20260927 11:13:30.310992 8118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:30.362870 8118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:30.364684 8118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:30.364842 8118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:30.365296 8118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:30.365355 8118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:30.365403 8118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:30.365432 8118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 11:13:30 dist-test-slave-9hq9 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:13:30.372605 8118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37525
I20260927 11:13:30.372680 8248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37525 every 8 connection(s)
I20260927 11:13:30.372989 8118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:13:30.374917 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.365809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43445 (local address 127.0.98.62:34979)
0927 11:13:30.365885 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:30.365888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:30.366021 (+ 133us) server_negotiation.cc:407] Connection header received
0927 11:13:30.366557 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:30.366559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:30.366572 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:30.366647 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:30.367192 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.368010 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.368998 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.369333 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.369431 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.372406 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:30.372408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:30.372408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:30.372416 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:30.373798 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.374259 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.374260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.374260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.374307 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.374484 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.374484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.374484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.374578 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:30.374622 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:30.374811 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 11:13:30.374862 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 11:13:30.375067 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.365740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.2:43445)
0927 11:13:30.365899 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:30.365935 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:30.366433 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:30.366706 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:30.366709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:30.366919 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:30.367092 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.367096 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.368138 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.368139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.368910 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.368912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.369442 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.369443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.369599 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.370165 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:30.370173 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:30.372266 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:30.373865 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:30.373869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:30.373870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:30.374198 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:30.374350 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:30.374350 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:30.374350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:30.374429 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:30.374628 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:30.374631 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:30.374726 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 11:13:30.374795 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 11:13:30.374893 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260927 11:13:30.375504 8249 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34979
I20260927 11:13:30.375595 8249 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:30.375748 8249 heartbeater.cc:511] Master 127.0.98.62:34979 requested a full tablet report, sending...
I20260927 11:13:30.376089 7906 ts_manager.cc:194] Registered new tserver with Master: d8f97afedb2f47858a02dcd706afcb75 (127.0.98.2:37525)
I20260927 11:13:30.376583 7906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43445
I20260927 11:13:30.380944 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.376484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50613 (local address 127.0.98.62:34979)
0927 11:13:30.376521 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:30.376522 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:30.376575 (+ 53us) server_negotiation.cc:407] Connection header received
0927 11:13:30.376641 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:30.376642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:30.376651 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:30.376700 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:30.377107 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.377526 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.378176 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.378336 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.378383 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.379160 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:30.379161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:30.379161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:30.379165 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:30.380176 (+ 1011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.380437 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.380437 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.380438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.380487 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.380599 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.380600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.380600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.380703 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:30.380744 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:30.380846 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 11:13:30.380896 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:13:30.380950 8255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.376409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.2:50613)
0927 11:13:30.376523 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:30.376534 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:30.376588 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:30.376771 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:30.376771 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:30.376957 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:30.377034 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.377036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.377603 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.377604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.378106 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.378108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.378395 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.378395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.378438 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.378847 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:30.378852 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:30.379098 (+ 246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:30.380239 (+ 1141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:30.380240 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:30.380240 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:30.380381 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:30.380529 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:30.380529 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:30.380529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:30.380567 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:30.380751 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:30.380751 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:30.380780 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 11:13:30.380828 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 11:13:30.380890 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":5,"thread_start_us":64,"threads_started":1}
I20260927 11:13:30.381702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 8118
I20260927 11:13:30.381829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:13:30.382030 392 external_mini_cluster.cc:1428] Setting key 4e4042b0bac3970eca7afbf5ed3634ec
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 11:13:30.402635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34979
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:30.479204 8258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:30.479386 8258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:30.479409 8258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:30.480535 8258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:30.480572 8258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:30.480592 8258 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 11:13:30.480633 8258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:30.482064 8258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:30.482287 8258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:30.482491 8258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:30.487288 8264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:30.487296 8266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:30.487296 8263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:30.487478 8258 server_base.cc:1034] running on GCE node
I20260927 11:13:30.487681 8258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:30.487926 8258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:30.489075 8258 hybrid_clock.cc:648] HybridClock initialized: now 1790507610489061 us; error 31 us; skew 500 ppm
Sep 27 11:13:30 dist-test-slave-9hq9 krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:30.491544 8258 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 11:13:30.491946 8258 webserver.cc:460] Webserver started at http://127.0.98.3:40213/ using document root <none> and password file <none>
I20260927 11:13:30.492116 8258 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:30.492174 8258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:30.492256 8258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:30.493227 8258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "2bea4a6f688147be81ffeef3903e5838"
format_stamp: "Formatted at 2026-09-27 11:13:30 on dist-test-slave-9hq9"
server_key: "b8be3492e723e7c9ceebf3e61600dea5"
server_key_iv: "94f8cc034646f98e4b7bc7937bdaa560"
server_key_version: "encryptionkey@0"
I20260927 11:13:30.493518 8258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2bea4a6f688147be81ffeef3903e5838"
format_stamp: "Formatted at 2026-09-27 11:13:30 on dist-test-slave-9hq9"
server_key: "b8be3492e723e7c9ceebf3e61600dea5"
server_key_iv: "94f8cc034646f98e4b7bc7937bdaa560"
server_key_version: "encryptionkey@0"
I20260927 11:13:30.494817 8258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:13:30.495422 8273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:30.495590 8258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:30.495702 8258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "2bea4a6f688147be81ffeef3903e5838"
format_stamp: "Formatted at 2026-09-27 11:13:30 on dist-test-slave-9hq9"
server_key: "b8be3492e723e7c9ceebf3e61600dea5"
server_key_iv: "94f8cc034646f98e4b7bc7937bdaa560"
server_key_version: "encryptionkey@0"
I20260927 11:13:30.495769 8258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:30.519896 8258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:30.521461 8258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:30.521581 8258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:30.521991 8258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:30.522022 8258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:30.522081 8258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:30.522116 8258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:30.528529 8258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36081
I20260927 11:13:30.528558 8388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36081 every 8 connection(s)
Sep 27 11:13:30 dist-test-slave-9hq9 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507610, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:13:30.528862 8258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:13:30.531343 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.522446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34093 (local address 127.0.98.62:34979)
0927 11:13:30.522483 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:30.522485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:30.522668 (+ 183us) server_negotiation.cc:407] Connection header received
0927 11:13:30.523089 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:30.523091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:30.523103 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:30.523176 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:30.523942 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.524755 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.526344 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.526649 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.526709 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.529146 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:30.529147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:30.529147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:30.529155 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:30.530331 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.530658 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.530659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.530659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.530704 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.530897 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.530897 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.530897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.530986 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:30.531034 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:30.531232 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 11:13:30.531286 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:13:30.531472 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.522445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.3:34093)
0927 11:13:30.522576 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:30.522604 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:30.522996 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:30.523226 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:30.523229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:30.523567 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:30.523847 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.523854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.524909 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.524910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.526249 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.526252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.526735 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.526736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.526870 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.527353 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:30.527360 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:30.529060 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:30.530393 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:30.530398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:30.530400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:30.530597 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:30.530757 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:30.530757 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:30.530758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:30.530846 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:30.531037 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:30.531041 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:30.531134 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 11:13:30.531208 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 11:13:30.531309 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":70,"threads_started":1}
I20260927 11:13:30.531949 8389 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34979
I20260927 11:13:30.532018 8389 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:30.532183 8389 heartbeater.cc:511] Master 127.0.98.62:34979 requested a full tablet report, sending...
I20260927 11:13:30.532555 7905 ts_manager.cc:194] Registered new tserver with Master: 2bea4a6f688147be81ffeef3903e5838 (127.0.98.3:36081)
I20260927 11:13:30.532866 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34093
I20260927 11:13:30.536691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 8258
I20260927 11:13:30.536783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:13:30.536947 392 external_mini_cluster.cc:1428] Setting key 92941eb8cd09cde3e4c1d9cc3c2af48f
I20260927 11:13:30.537639 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:13:30.537895 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.532962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60305 (local address 127.0.98.62:34979)
0927 11:13:30.533016 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:30.533017 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:30.533116 (+ 99us) server_negotiation.cc:407] Connection header received
0927 11:13:30.533181 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:30.533182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:30.533191 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:30.533247 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:30.533554 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.533981 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.534644 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.534805 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.534851 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.535489 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:30.535490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:30.535490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:30.535496 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:30.536862 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.537211 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.537212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.537212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.537263 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.537456 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.537457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.537457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.537574 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:30.537641 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:30.537756 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 11:13:30.537817 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:13:30.537906 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.532915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.98.3:60305)
0927 11:13:30.533062 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:30.533072 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:30.533132 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:30.533256 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:30.533257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:30.533436 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:30.533485 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.533487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.534107 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.534108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.534582 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.534583 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.534862 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.534863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.534904 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.535196 (+ 292us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:30.535199 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:30.535422 (+ 223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:30.536936 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:30.536937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:30.536937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:30.537129 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:30.537333 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:30.537334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:30.537334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:30.537387 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:30.537634 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:30.537634 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:30.537672 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 11:13:30.537733 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 11:13:30.537818 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
Sep 27 11:13:30 dist-test-slave-9hq9 krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507610, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260927 11:13:30.552498 7951 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.0.98.62:34979: Invalid argument: socket address is ambiguous
Sep 27 11:13:30 dist-test-slave-9hq9 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:30 dist-test-slave-9hq9 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 11:13:30.557433 7955 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:30.552672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36576 (local address 127.0.98.62:34979)
0927 11:13:30.552724 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:30.552726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:30.552735 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:13:30.552754 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:30.552755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:30.552763 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:30.552821 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:30.553342 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.553771 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.554514 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.554674 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.554721 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.557374 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36576: BlockingRecv error: recv got EOF from 127.0.0.1:36576 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 11:13:30 dist-test-slave-9hq9 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507610, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:13:30.563728 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.558152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36586 (local address 127.0.98.62:34979)
0927 11:13:30.558197 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:30.558199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:30.558262 (+ 63us) server_negotiation.cc:407] Connection header received
0927 11:13:30.558302 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:30.558303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:30.558310 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:30.558356 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:30.558784 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.559210 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.559810 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.559970 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.560020 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.561768 (+ 1748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:30.561770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:30.561770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:30.561775 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:30.562960 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.563239 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.563240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.563240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.563282 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:30.563418 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:30.563418 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:30.563418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:30.563494 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:30.563554 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:30.563632 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 11:13:30.563679 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 11:13:30.565096 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:13:30.566709 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:30.572573 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.568772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41965 (local address 127.0.0.1:44448)
0927 11:13:30.569049 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:30.569060 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:30.569110 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:30.569373 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:30.569375 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:30.569377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:30.569417 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.569418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.570732 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.570734 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.571497 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.571499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.572281 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.572282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.572359 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.572372 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 11:13:30.572445 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 11:13:30.572491 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":50,"threads_started":1}
I20260927 11:13:30.572587 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.568937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44448 (local address 127.0.98.1:41965)
0927 11:13:30.569133 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:30.569137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:30.569148 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:13:30.569192 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:30.569195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:30.569211 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:30.569301 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:30.569514 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.570627 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.571602 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.572197 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.572281 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.572362 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:30.572459 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 11:13:30.572494 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260927 11:13:30.573134 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.568951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37525 (local address 127.0.0.1:38912)
0927 11:13:30.569546 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:30.569555 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:30.569598 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:30.569778 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:30.569779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:30.569780 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:30.569837 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.569839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.570753 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.570753 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.572004 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.572006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.572984 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.572984 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.573033 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.573036 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:30.573066 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 11:13:30.573092 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":105,"thread_start_us":68,"threads_started":1}
I20260927 11:13:30.573479 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.568835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51468 (local address 127.0.98.3:36081)
0927 11:13:30.568973 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:30.568978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:30.569101 (+ 123us) server_negotiation.cc:407] Connection header received
0927 11:13:30.569211 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:30.569213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:30.569227 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:30.569289 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:30.569538 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.570605 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.572348 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.573040 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.573081 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.573143 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:30.573348 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 11:13:30.573383 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260927 11:13:30.573498 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfc0f50669e4424b3f5e3a230357bac (DEFAULT_TABLE table=test-table [id=9e5987ea063247248e5e69e5d63afcb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:30.573850 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfc0f50669e4424b3f5e3a230357bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:30.573866 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.568770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36081 (local address 127.0.0.1:51468)
0927 11:13:30.568997 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:30.569034 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:30.569126 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:30.569373 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:30.569376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:30.569387 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:30.569434 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.569435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.570694 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.570695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.572262 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.572264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.573203 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.573204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.573263 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.573266 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:30.573564 (+ 298us) client_negotiation.cc:240] Negotiation successful
0927 11:13:30.573606 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":85,"thread_start_us":67,"threads_started":1}
I20260927 11:13:30.574273 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.569147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.0.98.2:37525)
0927 11:13:30.569307 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:30.569312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:30.569652 (+ 340us) server_negotiation.cc:407] Connection header received
0927 11:13:30.569693 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:30.569695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:30.569711 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:30.569782 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:30.569891 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.570643 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.572087 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.572817 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.572904 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.574120 (+ 1216us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:30.574145 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 11:13:30.574183 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260927 11:13:30.574489 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfc0f50669e4424b3f5e3a230357bac (DEFAULT_TABLE table=test-table [id=9e5987ea063247248e5e69e5d63afcb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:30.574803 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfc0f50669e4424b3f5e3a230357bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:30.574872 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 6cfc0f50669e4424b3f5e3a230357bac (DEFAULT_TABLE table=test-table [id=9e5987ea063247248e5e69e5d63afcb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:30.575184 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cfc0f50669e4424b3f5e3a230357bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:30.577066 8417 tablet_bootstrap.cc:492] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0: Bootstrap starting.
I20260927 11:13:30.577590 8417 tablet_bootstrap.cc:654] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:30.577800 8417 log.cc:824] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:30.578543 8417 tablet_bootstrap.cc:492] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0: No bootstrap required, opened a new log
I20260927 11:13:30.578616 8417 ts_tablet_manager.cc:1397] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:30.578655 8419 tablet_bootstrap.cc:492] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838: Bootstrap starting.
I20260927 11:13:30.579218 8419 tablet_bootstrap.cc:654] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:30.579491 8419 log.cc:824] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:30.579777 8420 tablet_bootstrap.cc:492] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75: Bootstrap starting.
I20260927 11:13:30.580186 8419 tablet_bootstrap.cc:492] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838: No bootstrap required, opened a new log
I20260927 11:13:30.580252 8419 ts_tablet_manager.cc:1397] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:30.580191 8417 raft_consensus.cc:348] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } }
I20260927 11:13:30.580365 8420 tablet_bootstrap.cc:654] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:30.580385 8417 raft_consensus.cc:374] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:30.580431 8417 raft_consensus.cc:729] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8210201d68e843cc90b1d2e08323e5c0, State: Initialized, Role: FOLLOWER
I20260927 11:13:30.580554 8417 consensus_queue.cc:260] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } }
I20260927 11:13:30.580668 8420 log.cc:824] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:30.580822 8417 ts_tablet_manager.cc:1428] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:13:30.580842 8109 heartbeater.cc:503] Master 127.0.98.62:34979 was elected leader, sending a full tablet report...
I20260927 11:13:30.581311 8420 tablet_bootstrap.cc:492] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75: No bootstrap required, opened a new log
I20260927 11:13:30.581377 8420 ts_tablet_manager.cc:1397] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:30.582298 8419 raft_consensus.cc:348] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } }
I20260927 11:13:30.582445 8419 raft_consensus.cc:374] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:30.582477 8419 raft_consensus.cc:729] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bea4a6f688147be81ffeef3903e5838, State: Initialized, Role: FOLLOWER
I20260927 11:13:30.582585 8419 consensus_queue.cc:260] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } }
I20260927 11:13:30.582818 8419 ts_tablet_manager.cc:1428] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:13:30.582844 8389 heartbeater.cc:503] Master 127.0.98.62:34979 was elected leader, sending a full tablet report...
I20260927 11:13:30.583264 8420 raft_consensus.cc:348] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } }
I20260927 11:13:30.583427 8420 raft_consensus.cc:374] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:30.583472 8420 raft_consensus.cc:729] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f97afedb2f47858a02dcd706afcb75, State: Initialized, Role: FOLLOWER
I20260927 11:13:30.583585 8420 consensus_queue.cc:260] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } }
I20260927 11:13:30.583837 8420 ts_tablet_manager.cc:1428] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:13:30.583853 8249 heartbeater.cc:503] Master 127.0.98.62:34979 was elected leader, sending a full tablet report...
W20260927 11:13:30.623627 8250 tablet.cc:2369] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:13:30.686484 8110 tablet.cc:2369] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:13:30.779608 8390 tablet.cc:2369] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:13:30.795604 8425 raft_consensus.cc:482] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:30.795732 8425 raft_consensus.cc:504] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } }
I20260927 11:13:30.796080 8425 leader_election.cc:288] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bea4a6f688147be81ffeef3903e5838 (127.0.98.3:36081), 8210201d68e843cc90b1d2e08323e5c0 (127.0.98.1:41965)
I20260927 11:13:30.799809 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.796345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59997 (local address 127.0.98.1:41965)
0927 11:13:30.796390 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:30.796392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:30.796406 (+ 14us) server_negotiation.cc:407] Connection header received
0927 11:13:30.796452 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:30.796454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:30.796466 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:30.796518 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:30.796701 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.797454 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.798878 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.799467 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.799527 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.799536 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:30.799708 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 11:13:30.799743 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:13:30.799808 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.796348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43873 (local address 127.0.98.3:36081)
0927 11:13:30.796396 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:30.796399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:30.796685 (+ 286us) server_negotiation.cc:407] Connection header received
0927 11:13:30.796713 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:30.796714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:30.796726 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:30.796786 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:30.796907 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.797573 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.798877 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.799418 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.799500 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.799512 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:30.799708 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 11:13:30.799743 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 11:13:30.799872 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.796258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41965 (local address 127.0.98.2:59997)
0927 11:13:30.796320 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:30.796334 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:30.796386 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:30.796550 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:30.796551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:30.796553 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:30.796625 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.796627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.797528 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.797529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.798792 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.798794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.799533 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.799533 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.799622 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.799629 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 11:13:30.799689 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 11:13:30.799734 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":46}
I20260927 11:13:30.800041 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.796258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36081 (local address 127.0.98.2:43873)
0927 11:13:30.796592 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:30.796604 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:30.796660 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:30.796789 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:30.796790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:30.796799 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:30.796851 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.796852 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.797649 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.797649 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.798792 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:30.798794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.799528 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.799529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:30.799622 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.799630 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 11:13:30.799688 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 11:13:30.799734 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":164,"thread_start_us":111,"threads_started":1}
I20260927 11:13:30.800228 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfc0f50669e4424b3f5e3a230357bac" candidate_uuid: "d8f97afedb2f47858a02dcd706afcb75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8210201d68e843cc90b1d2e08323e5c0" is_pre_election: true
I20260927 11:13:30.800395 8063 raft_consensus.cc:2454] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f97afedb2f47858a02dcd706afcb75 in term 0.
I20260927 11:13:30.800370 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfc0f50669e4424b3f5e3a230357bac" candidate_uuid: "d8f97afedb2f47858a02dcd706afcb75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bea4a6f688147be81ffeef3903e5838" is_pre_election: true
I20260927 11:13:30.800482 8343 raft_consensus.cc:2454] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f97afedb2f47858a02dcd706afcb75 in term 0.
I20260927 11:13:30.800917 8134 leader_election.cc:302] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8210201d68e843cc90b1d2e08323e5c0, d8f97afedb2f47858a02dcd706afcb75; no voters:
I20260927 11:13:30.801096 8425 raft_consensus.cc:2781] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:30.801151 8425 raft_consensus.cc:482] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:30.801178 8425 raft_consensus.cc:3037] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:30.801846 8425 raft_consensus.cc:504] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } }
I20260927 11:13:30.801995 8425 leader_election.cc:288] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [CANDIDATE]: Term 1 election: Requested vote from peers 2bea4a6f688147be81ffeef3903e5838 (127.0.98.3:36081), 8210201d68e843cc90b1d2e08323e5c0 (127.0.98.1:41965)
I20260927 11:13:30.802131 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfc0f50669e4424b3f5e3a230357bac" candidate_uuid: "d8f97afedb2f47858a02dcd706afcb75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bea4a6f688147be81ffeef3903e5838"
I20260927 11:13:30.802139 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cfc0f50669e4424b3f5e3a230357bac" candidate_uuid: "d8f97afedb2f47858a02dcd706afcb75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8210201d68e843cc90b1d2e08323e5c0"
I20260927 11:13:30.802208 8343 raft_consensus.cc:3037] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:30.802230 8063 raft_consensus.cc:3037] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:30.802873 8063 raft_consensus.cc:2454] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f97afedb2f47858a02dcd706afcb75 in term 1.
I20260927 11:13:30.802891 8343 raft_consensus.cc:2454] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f97afedb2f47858a02dcd706afcb75 in term 1.
I20260927 11:13:30.803051 8134 leader_election.cc:302] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8210201d68e843cc90b1d2e08323e5c0, d8f97afedb2f47858a02dcd706afcb75; no voters:
I20260927 11:13:30.803159 8425 raft_consensus.cc:2781] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:30.803274 8425 raft_consensus.cc:686] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [term 1 LEADER]: Becoming Leader. State: Replica: d8f97afedb2f47858a02dcd706afcb75, State: Running, Role: LEADER
I20260927 11:13:30.803383 8425 consensus_queue.cc:237] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } }
I20260927 11:13:30.804119 7905 catalog_manager.cc:5495] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8f97afedb2f47858a02dcd706afcb75 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d8f97afedb2f47858a02dcd706afcb75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } health_report { overall_health: HEALTHY } } }
I20260927 11:13:30.854772 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:30.852806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.0.98.2:37525)
0927 11:13:30.852863 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:30.852865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:30.852871 (+ 6us) server_negotiation.cc:407] Connection header received
0927 11:13:30.852894 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:30.852895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:30.852910 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:30.852989 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:30.853090 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.853524 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.854240 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:30.854396 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:30.854438 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:30.854554 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:30.854633 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:30.854683 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:30.854714 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 11:13:30.854729 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 11:13:30.856266 8063 raft_consensus.cc:1261] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Refusing update from remote peer d8f97afedb2f47858a02dcd706afcb75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:30.856277 8343 raft_consensus.cc:1261] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Refusing update from remote peer d8f97afedb2f47858a02dcd706afcb75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:30.856564 8427 consensus_queue.cc:1035] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:30.856688 8425 consensus_queue.cc:1035] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:30.858980 8430 mvcc.cc:205] Tried to move back new op lower bound from 7333919174065004544 to 7333919173851299840. Current Snapshot: MvccSnapshot[applied={T|T < 7333919174065004544}]
I20260927 11:13:30.859201 8432 mvcc.cc:205] Tried to move back new op lower bound from 7333919174065004544 to 7333919173851299840. Current Snapshot: MvccSnapshot[applied={T|T < 7333919174065004544}]
I20260927 11:13:30.859773 8431 mvcc.cc:205] Tried to move back new op lower bound from 7333919174065004544 to 7333919173851299840. Current Snapshot: MvccSnapshot[applied={T|T < 7333919174065004544}]
I20260927 11:13:30.861302 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:30.861375 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:30.862035 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42: Sending DeleteTablet for 3 replicas of tablet 6cfc0f50669e4424b3f5e3a230357bac
I20260927 11:13:30.862388 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cfc0f50669e4424b3f5e3a230357bac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38912
I20260927 11:13:30.862386 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cfc0f50669e4424b3f5e3a230357bac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44448
I20260927 11:13:30.862545 8440 tablet_replica.cc:330] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0: stopping tablet replica
I20260927 11:13:30.862639 8440 raft_consensus.cc:2229] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:30.862596 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:13:30.862730 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:30.862741 8440 raft_consensus.cc:2258] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:30.862741 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cfc0f50669e4424b3f5e3a230357bac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51468
I20260927 11:13:30.862912 8441 tablet_replica.cc:330] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75: stopping tablet replica
I20260927 11:13:30.863018 8441 raft_consensus.cc:2229] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:13:30.863029 8442 tablet_replica.cc:330] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838: stopping tablet replica
I20260927 11:13:30.863096 8442 raft_consensus.cc:2229] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:30.863111 8441 raft_consensus.cc:2258] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:30.863194 8442 raft_consensus.cc:2258] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:30.863535 8441 ts_tablet_manager.cc:1905] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:30.863534 8442 ts_tablet_manager.cc:1905] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:30.864532 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 7784dc12f60a4638b4763f0bae489078 (DEFAULT_TABLE table=test-table [id=de3afa281d4b4f0abd4ec306ad2f81d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:30.864636 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7784dc12f60a4638b4763f0bae489078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:30.865266 8441 ts_tablet_manager.cc:1918] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:30.865266 8442 ts_tablet_manager.cc:1918] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:30.865329 8441 log.cc:1196] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/6cfc0f50669e4424b3f5e3a230357bac
I20260927 11:13:30.865329 8442 log.cc:1196] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/6cfc0f50669e4424b3f5e3a230357bac
I20260927 11:13:30.865583 8441 ts_tablet_manager.cc:1939] T 6cfc0f50669e4424b3f5e3a230357bac P d8f97afedb2f47858a02dcd706afcb75: Deleting consensus metadata
I20260927 11:13:30.865583 8442 ts_tablet_manager.cc:1939] T 6cfc0f50669e4424b3f5e3a230357bac P 2bea4a6f688147be81ffeef3903e5838: Deleting consensus metadata
I20260927 11:13:30.866029 8417 tablet_bootstrap.cc:492] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0: Bootstrap starting.
I20260927 11:13:30.866050 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 7784dc12f60a4638b4763f0bae489078 (DEFAULT_TABLE table=test-table [id=de3afa281d4b4f0abd4ec306ad2f81d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:30.866149 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7784dc12f60a4638b4763f0bae489078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:30.866340 8417 tablet_bootstrap.cc:654] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:30.866791 8417 tablet_bootstrap.cc:492] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0: No bootstrap required, opened a new log
I20260927 11:13:30.866817 8417 ts_tablet_manager.cc:1397] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:30.866918 8417 raft_consensus.cc:348] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:30.866952 8417 raft_consensus.cc:374] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:30.866989 8417 raft_consensus.cc:729] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8210201d68e843cc90b1d2e08323e5c0, State: Initialized, Role: FOLLOWER
I20260927 11:13:30.867053 8417 consensus_queue.cc:260] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:30.867148 8417 ts_tablet_manager.cc:1428] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:30.867190 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 7784dc12f60a4638b4763f0bae489078 (DEFAULT_TABLE table=test-table [id=de3afa281d4b4f0abd4ec306ad2f81d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:30.867274 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7784dc12f60a4638b4763f0bae489078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:30.868285 8420 tablet_bootstrap.cc:492] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75: Bootstrap starting.
I20260927 11:13:30.868291 8419 tablet_bootstrap.cc:492] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838: Bootstrap starting.
I20260927 11:13:30.868623 8419 tablet_bootstrap.cc:654] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:30.868620 8420 tablet_bootstrap.cc:654] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:30.868923 7891 catalog_manager.cc:4841] TS 2bea4a6f688147be81ffeef3903e5838 (127.0.98.3:36081): tablet 6cfc0f50669e4424b3f5e3a230357bac (table test-table [id=9e5987ea063247248e5e69e5d63afcb2]) successfully deleted
I20260927 11:13:30.869005 7891 catalog_manager.cc:4841] TS d8f97afedb2f47858a02dcd706afcb75 (127.0.98.2:37525): tablet 6cfc0f50669e4424b3f5e3a230357bac (table test-table [id=9e5987ea063247248e5e69e5d63afcb2]) successfully deleted
I20260927 11:13:30.869175 8419 tablet_bootstrap.cc:492] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838: No bootstrap required, opened a new log
I20260927 11:13:30.869206 8419 ts_tablet_manager.cc:1397] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:30.869299 8419 raft_consensus.cc:348] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:30.869334 8419 raft_consensus.cc:374] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:30.869345 8419 raft_consensus.cc:729] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bea4a6f688147be81ffeef3903e5838, State: Initialized, Role: FOLLOWER
I20260927 11:13:30.869398 8419 consensus_queue.cc:260] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:30.869472 8419 ts_tablet_manager.cc:1428] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:30.870222 8440 ts_tablet_manager.cc:1905] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:30.872012 8440 ts_tablet_manager.cc:1918] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:30.872079 8440 log.cc:1196] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/wal/wals/6cfc0f50669e4424b3f5e3a230357bac
I20260927 11:13:30.872339 8440 ts_tablet_manager.cc:1939] T 6cfc0f50669e4424b3f5e3a230357bac P 8210201d68e843cc90b1d2e08323e5c0: Deleting consensus metadata
I20260927 11:13:30.872714 7890 catalog_manager.cc:4841] TS 8210201d68e843cc90b1d2e08323e5c0 (127.0.98.1:41965): tablet 6cfc0f50669e4424b3f5e3a230357bac (table test-table [id=9e5987ea063247248e5e69e5d63afcb2]) successfully deleted
I20260927 11:13:30.872859 8420 tablet_bootstrap.cc:492] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75: No bootstrap required, opened a new log
I20260927 11:13:30.872908 8420 ts_tablet_manager.cc:1397] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 11:13:30.873049 8420 raft_consensus.cc:348] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:30.873111 8420 raft_consensus.cc:374] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:30.873131 8420 raft_consensus.cc:729] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f97afedb2f47858a02dcd706afcb75, State: Initialized, Role: FOLLOWER
I20260927 11:13:30.873168 8420 consensus_queue.cc:260] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:30.873256 8420 ts_tablet_manager.cc:1428] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260927 11:13:31.011169 7947 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.98.62:34979
I20260927 11:13:31.046200 8422 raft_consensus.cc:482] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:31.046321 8422 raft_consensus.cc:504] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.046650 8422 leader_election.cc:288] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8f97afedb2f47858a02dcd706afcb75 (127.0.98.2:37525), 2bea4a6f688147be81ffeef3903e5838 (127.0.98.3:36081)
I20260927 11:13:31.049667 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.046923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59013 (local address 127.0.98.2:37525)
0927 11:13:31.046995 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:31.046997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:31.047052 (+ 55us) server_negotiation.cc:407] Connection header received
0927 11:13:31.047129 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:31.047130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:31.047139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:31.047197 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:31.047356 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.047829 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.048786 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.049304 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.049385 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.049395 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:31.049550 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 11:13:31.049584 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:13:31.049657 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.046837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37525 (local address 127.0.98.1:59013)
0927 11:13:31.046982 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:31.046994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:31.047067 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:31.047204 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:31.047205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:31.047206 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:31.047280 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.047283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.047916 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.047917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.048694 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.048696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.049384 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.049385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.049457 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.049461 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:31.049519 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 11:13:31.049557 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260927 11:13:31.049734 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.047093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33889 (local address 127.0.98.3:36081)
0927 11:13:31.047137 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:31.047139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:31.047251 (+ 112us) server_negotiation.cc:407] Connection header received
0927 11:13:31.047315 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:31.047316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:31.047325 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:31.047394 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:31.047543 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.047985 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.048979 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.049429 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.049483 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.049490 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:31.049631 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 11:13:31.049667 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:13:31.049947 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.047031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36081 (local address 127.0.98.1:33889)
0927 11:13:31.047171 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:31.047180 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:31.047259 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:31.047396 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:31.047397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:31.047398 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:31.047465 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.047466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.048073 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.048074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.048841 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.048843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.049505 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.049506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.049565 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.049567 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:13:31.049618 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 11:13:31.049645 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260927 11:13:31.049993 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7784dc12f60a4638b4763f0bae489078" candidate_uuid: "8210201d68e843cc90b1d2e08323e5c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8f97afedb2f47858a02dcd706afcb75" is_pre_election: true
I20260927 11:13:31.050155 8201 raft_consensus.cc:2454] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8210201d68e843cc90b1d2e08323e5c0 in term 0.
I20260927 11:13:31.050371 7995 leader_election.cc:302] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8210201d68e843cc90b1d2e08323e5c0, d8f97afedb2f47858a02dcd706afcb75; no voters:
I20260927 11:13:31.050418 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7784dc12f60a4638b4763f0bae489078" candidate_uuid: "8210201d68e843cc90b1d2e08323e5c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bea4a6f688147be81ffeef3903e5838" is_pre_election: true
I20260927 11:13:31.050490 8422 raft_consensus.cc:2781] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:31.050508 8343 raft_consensus.cc:2454] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8210201d68e843cc90b1d2e08323e5c0 in term 0.
I20260927 11:13:31.050525 8422 raft_consensus.cc:482] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:31.050545 8422 raft_consensus.cc:3037] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:31.051215 8422 raft_consensus.cc:504] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.051339 8422 leader_election.cc:288] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [CANDIDATE]: Term 1 election: Requested vote from peers d8f97afedb2f47858a02dcd706afcb75 (127.0.98.2:37525), 2bea4a6f688147be81ffeef3903e5838 (127.0.98.3:36081)
I20260927 11:13:31.051481 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7784dc12f60a4638b4763f0bae489078" candidate_uuid: "8210201d68e843cc90b1d2e08323e5c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8f97afedb2f47858a02dcd706afcb75"
I20260927 11:13:31.051481 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7784dc12f60a4638b4763f0bae489078" candidate_uuid: "8210201d68e843cc90b1d2e08323e5c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bea4a6f688147be81ffeef3903e5838"
I20260927 11:13:31.051558 8343 raft_consensus.cc:3037] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:31.051558 8201 raft_consensus.cc:3037] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:31.052292 8201 raft_consensus.cc:2454] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8210201d68e843cc90b1d2e08323e5c0 in term 1.
I20260927 11:13:31.052292 8343 raft_consensus.cc:2454] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8210201d68e843cc90b1d2e08323e5c0 in term 1.
I20260927 11:13:31.052464 7995 leader_election.cc:302] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8210201d68e843cc90b1d2e08323e5c0, d8f97afedb2f47858a02dcd706afcb75; no voters:
I20260927 11:13:31.052556 8422 raft_consensus.cc:2781] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:31.052709 8422 raft_consensus.cc:686] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [term 1 LEADER]: Becoming Leader. State: Replica: 8210201d68e843cc90b1d2e08323e5c0, State: Running, Role: LEADER
I20260927 11:13:31.052783 8422 consensus_queue.cc:237] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.053411 7906 catalog_manager.cc:5495] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8210201d68e843cc90b1d2e08323e5c0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8210201d68e843cc90b1d2e08323e5c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } health_report { overall_health: UNKNOWN } } }
Sep 27 11:13:31 dist-test-slave-9hq9 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507609, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:13:31.116722 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.110782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.0.98.62:34979)
0927 11:13:31.110918 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:31.110920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:31.110928 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:13:31.110945 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:31.110945 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:31.110954 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:31.111019 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:31.111408 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.111870 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.112499 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.112674 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.112721 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.114678 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:31.114679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:31.114679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:31.114688 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:31.115829 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:31.116150 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:31.116150 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:31.116151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:31.116197 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:31.116349 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:31.116350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:31.116350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:31.116446 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:31.116493 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:31.116597 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 11:13:31.116660 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260927 11:13:31.116844 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.110688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34979 (local address 127.0.0.1:36588)
0927 11:13:31.110819 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:31.110830 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:31.110890 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:31.111036 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:31.111037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:31.111308 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:31.111347 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.111348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.111930 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.111931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.112439 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.112440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.112732 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.112733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.112768 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.113255 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:31.113259 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:31.114558 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:31.115894 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:31.115899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:31.115901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:31.116090 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:31.116251 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:31.116252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:31.116252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:31.116302 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:31.116495 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:31.116499 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:31.116535 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 11:13:31.116584 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 11:13:31.116722 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":61,"thread_start_us":73,"threads_started":1}
I20260927 11:13:31.118727 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36588:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 11:13:31.118947 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:31.120200 8043 tablet_service.cc:1467] Processing CreateTablet for tablet c08658fade064e8daaa9ad0146e23c0a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=855bd5c45e244743af510cd414c8cb14]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:13:31.120201 8183 tablet_service.cc:1467] Processing CreateTablet for tablet c08658fade064e8daaa9ad0146e23c0a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=855bd5c45e244743af510cd414c8cb14]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:13:31.120347 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c08658fade064e8daaa9ad0146e23c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:31.120353 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c08658fade064e8daaa9ad0146e23c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:31.121395 8417 tablet_bootstrap.cc:492] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0: Bootstrap starting.
I20260927 11:13:31.121465 8420 tablet_bootstrap.cc:492] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75: Bootstrap starting.
I20260927 11:13:31.121765 8420 tablet_bootstrap.cc:654] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:31.121786 8417 tablet_bootstrap.cc:654] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:31.122140 8323 tablet_service.cc:1467] Processing CreateTablet for tablet c08658fade064e8daaa9ad0146e23c0a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=855bd5c45e244743af510cd414c8cb14]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 11:13:31.122272 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c08658fade064e8daaa9ad0146e23c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:31.122362 8417 tablet_bootstrap.cc:492] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0: No bootstrap required, opened a new log
I20260927 11:13:31.122400 8417 ts_tablet_manager.cc:1397] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:31.122522 8417 raft_consensus.cc:348] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.122570 8417 raft_consensus.cc:374] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:31.122582 8417 raft_consensus.cc:729] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8210201d68e843cc90b1d2e08323e5c0, State: Initialized, Role: FOLLOWER
I20260927 11:13:31.122651 8417 consensus_queue.cc:260] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.122745 8417 ts_tablet_manager.cc:1428] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:31.122776 8422 tablet_replica.cc:439] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } }
I20260927 11:13:31.122840 8422 tablet_replica.cc:442] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:31.123598 8419 tablet_bootstrap.cc:492] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838: Bootstrap starting.
I20260927 11:13:31.123750 8420 tablet_bootstrap.cc:492] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75: No bootstrap required, opened a new log
I20260927 11:13:31.123798 8420 ts_tablet_manager.cc:1397] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:31.123929 8420 raft_consensus.cc:348] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.123982 8420 raft_consensus.cc:374] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:31.123996 8420 raft_consensus.cc:729] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f97afedb2f47858a02dcd706afcb75, State: Initialized, Role: FOLLOWER
I20260927 11:13:31.124032 8419 tablet_bootstrap.cc:654] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:31.124045 8420 consensus_queue.cc:260] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.124136 8420 ts_tablet_manager.cc:1428] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:31.124251 8425 tablet_replica.cc:439] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } }
I20260927 11:13:31.124318 8425 tablet_replica.cc:442] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:31.124680 8419 tablet_bootstrap.cc:492] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838: No bootstrap required, opened a new log
I20260927 11:13:31.124729 8419 ts_tablet_manager.cc:1397] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:31.124907 8419 raft_consensus.cc:348] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.124997 8419 raft_consensus.cc:374] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:31.125030 8419 raft_consensus.cc:729] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bea4a6f688147be81ffeef3903e5838, State: Initialized, Role: FOLLOWER
I20260927 11:13:31.125072 8419 consensus_queue.cc:260] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.125166 8419 ts_tablet_manager.cc:1428] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:31.125207 8424 tablet_replica.cc:439] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } }
I20260927 11:13:31.125304 8424 tablet_replica.cc:442] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:31.126852 8422 raft_consensus.cc:482] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:31.126921 8422 raft_consensus.cc:504] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.127030 8422 leader_election.cc:288] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8f97afedb2f47858a02dcd706afcb75 (127.0.98.2:37525), 2bea4a6f688147be81ffeef3903e5838 (127.0.98.3:36081)
I20260927 11:13:31.127197 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08658fade064e8daaa9ad0146e23c0a" candidate_uuid: "8210201d68e843cc90b1d2e08323e5c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8f97afedb2f47858a02dcd706afcb75" is_pre_election: true
I20260927 11:13:31.127266 8201 raft_consensus.cc:2454] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8210201d68e843cc90b1d2e08323e5c0 in term 0.
I20260927 11:13:31.127249 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08658fade064e8daaa9ad0146e23c0a" candidate_uuid: "8210201d68e843cc90b1d2e08323e5c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bea4a6f688147be81ffeef3903e5838" is_pre_election: true
I20260927 11:13:31.127302 8343 raft_consensus.cc:2454] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8210201d68e843cc90b1d2e08323e5c0 in term 0.
I20260927 11:13:31.127445 7995 leader_election.cc:302] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bea4a6f688147be81ffeef3903e5838, 8210201d68e843cc90b1d2e08323e5c0; no voters:
I20260927 11:13:31.127549 8422 raft_consensus.cc:2781] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:31.127614 8422 raft_consensus.cc:482] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:31.127643 8422 raft_consensus.cc:3037] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:31.128129 8422 raft_consensus.cc:504] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.128245 8422 leader_election.cc:288] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [CANDIDATE]: Term 1 election: Requested vote from peers d8f97afedb2f47858a02dcd706afcb75 (127.0.98.2:37525), 2bea4a6f688147be81ffeef3903e5838 (127.0.98.3:36081)
I20260927 11:13:31.128371 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08658fade064e8daaa9ad0146e23c0a" candidate_uuid: "8210201d68e843cc90b1d2e08323e5c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8f97afedb2f47858a02dcd706afcb75"
I20260927 11:13:31.128412 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c08658fade064e8daaa9ad0146e23c0a" candidate_uuid: "8210201d68e843cc90b1d2e08323e5c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bea4a6f688147be81ffeef3903e5838"
I20260927 11:13:31.128433 8201 raft_consensus.cc:3037] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:31.128464 8343 raft_consensus.cc:3037] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:31.128994 8343 raft_consensus.cc:2454] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8210201d68e843cc90b1d2e08323e5c0 in term 1.
I20260927 11:13:31.129127 8201 raft_consensus.cc:2454] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8210201d68e843cc90b1d2e08323e5c0 in term 1.
I20260927 11:13:31.129136 7995 leader_election.cc:302] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bea4a6f688147be81ffeef3903e5838, 8210201d68e843cc90b1d2e08323e5c0; no voters:
I20260927 11:13:31.129232 8422 raft_consensus.cc:2781] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:31.129283 8422 raft_consensus.cc:686] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [term 1 LEADER]: Becoming Leader. State: Replica: 8210201d68e843cc90b1d2e08323e5c0, State: Running, Role: LEADER
I20260927 11:13:31.129324 8422 consensus_queue.cc:237] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.129601 8448 tablet_replica.cc:439] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0: TxnStatusTablet state changed. Reason: New leader 8210201d68e843cc90b1d2e08323e5c0. Latest consensus state: current_term: 1 leader_uuid: "8210201d68e843cc90b1d2e08323e5c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } }
I20260927 11:13:31.129698 8448 tablet_replica.cc:442] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:13:31.129855 8461 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 11:13:31.129841 7904 catalog_manager.cc:5495] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8210201d68e843cc90b1d2e08323e5c0 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8210201d68e843cc90b1d2e08323e5c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } health_report { overall_health: UNKNOWN } } }
I20260927 11:13:31.129959 8461 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 11:13:31.136061 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.133385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41965 (local address 127.0.0.1:44458)
0927 11:13:31.133526 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:31.133536 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:31.133580 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:31.133733 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:31.133734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:31.133735 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:13:31.133769 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.133770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.134657 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.134657 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.135190 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.135191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.135657 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.135658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.135692 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.135704 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:13:31.135943 (+ 239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:13:31.135948 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:13:31.135982 (+ 34us) client_negotiation.cc:240] Negotiation successful
0927 11:13:31.136016 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260927 11:13:31.136081 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.133467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44458 (local address 127.0.98.1:41965)
0927 11:13:31.133511 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:31.133513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:31.133567 (+ 54us) server_negotiation.cc:407] Connection header received
0927 11:13:31.133636 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:31.133637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:31.133654 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:31.133726 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:31.133810 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.134568 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.135267 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.135565 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.135656 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.135773 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:31.135879 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:31.135954 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:31.135996 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 11:13:31.136017 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:13:31.137064 8343 raft_consensus.cc:1261] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Refusing update from remote peer 8210201d68e843cc90b1d2e08323e5c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:31.137121 8201 raft_consensus.cc:1261] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Refusing update from remote peer 8210201d68e843cc90b1d2e08323e5c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:31.137295 8422 consensus_queue.cc:1035] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:31.137390 8448 consensus_queue.cc:1035] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:31.137880 8424 tablet_replica.cc:439] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838: TxnStatusTablet state changed. Reason: New leader 8210201d68e843cc90b1d2e08323e5c0. Latest consensus state: current_term: 1 leader_uuid: "8210201d68e843cc90b1d2e08323e5c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } }
I20260927 11:13:31.138005 8424 tablet_replica.cc:442] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:31.138309 8425 tablet_replica.cc:439] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75: TxnStatusTablet state changed. Reason: New leader 8210201d68e843cc90b1d2e08323e5c0. Latest consensus state: current_term: 1 leader_uuid: "8210201d68e843cc90b1d2e08323e5c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } }
I20260927 11:13:31.138394 8425 tablet_replica.cc:442] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:31.138883 8448 tablet_replica.cc:439] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8210201d68e843cc90b1d2e08323e5c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } }
I20260927 11:13:31.138967 8448 tablet_replica.cc:442] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:13:31.139017 8448 tablet_replica.cc:439] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8210201d68e843cc90b1d2e08323e5c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } }
I20260927 11:13:31.139055 8448 tablet_replica.cc:442] T c08658fade064e8daaa9ad0146e23c0a P 8210201d68e843cc90b1d2e08323e5c0: This TxnStatusTablet replica's current role is: LEADER
I20260927 11:13:31.139099 8424 tablet_replica.cc:439] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8210201d68e843cc90b1d2e08323e5c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } }
I20260927 11:13:31.139173 8424 tablet_replica.cc:442] T c08658fade064e8daaa9ad0146e23c0a P 2bea4a6f688147be81ffeef3903e5838: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:31.139788 8425 tablet_replica.cc:439] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8210201d68e843cc90b1d2e08323e5c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } } }
I20260927 11:13:31.139863 8425 tablet_replica.cc:442] T c08658fade064e8daaa9ad0146e23c0a P d8f97afedb2f47858a02dcd706afcb75: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 11:13:31.142799 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.140581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.0.98.1:41965)
0927 11:13:31.140611 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:31.140613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:31.140620 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:13:31.140679 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:31.140679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:31.140687 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:31.140728 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:31.140854 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.141270 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.142281 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.142441 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.142491 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.142639 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:31.142659 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:31.142686 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:31.142747 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 11:13:31.142760 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 11:13:31.147795 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.145108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36065 (local address 127.0.98.1:41965)
0927 11:13:31.145162 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:31.145165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:31.145659 (+ 494us) server_negotiation.cc:407] Connection header received
0927 11:13:31.145717 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:31.145718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:31.145726 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:31.145777 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:31.145919 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.146390 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.147152 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.147328 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.147405 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.147565 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:31.147598 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:31.147655 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:31.147713 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 11:13:31.147725 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 11:13:31.147884 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.145068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41965 (local address 127.0.98.1:36065)
0927 11:13:31.145583 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:31.145600 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:31.145672 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:31.145780 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:31.145780 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:31.145793 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:13:31.145858 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.145859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.146478 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.146479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.147059 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.147061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.147406 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.147407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.147462 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.147480 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:13:31.147656 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:13:31.147658 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:13:31.147710 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 11:13:31.147742 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":454,"threads_started":1}
I20260927 11:13:31.149578 8201 raft_consensus.cc:1261] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Refusing update from remote peer 8210201d68e843cc90b1d2e08323e5c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:31.149612 8343 raft_consensus.cc:1261] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Refusing update from remote peer 8210201d68e843cc90b1d2e08323e5c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:31.149794 8466 consensus_queue.cc:1035] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:31.149884 8448 consensus_queue.cc:1035] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:31.160734 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:31.160817 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:31.161351 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42: Sending DeleteTablet for 3 replicas of tablet 7784dc12f60a4638b4763f0bae489078
I20260927 11:13:31.161550 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 7784dc12f60a4638b4763f0bae489078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44448
I20260927 11:13:31.161578 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 7784dc12f60a4638b4763f0bae489078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38912
I20260927 11:13:31.161610 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 7784dc12f60a4638b4763f0bae489078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51468
I20260927 11:13:31.161664 8440 tablet_replica.cc:330] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0: stopping tablet replica
I20260927 11:13:31.161669 8441 tablet_replica.cc:330] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75: stopping tablet replica
I20260927 11:13:31.161684 8442 tablet_replica.cc:330] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838: stopping tablet replica
I20260927 11:13:31.161711 8440 raft_consensus.cc:2229] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:13:31.161720 8442 raft_consensus.cc:2229] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:31.161727 8441 raft_consensus.cc:2229] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:31.161754 8442 raft_consensus.cc:2258] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:31.161757 8441 raft_consensus.cc:2258] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:31.161783 8440 raft_consensus.cc:2258] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:31.162075 8442 ts_tablet_manager.cc:1905] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:31.162248 8440 ts_tablet_manager.cc:1905] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:31.163843 8441 ts_tablet_manager.cc:1905] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:31.165288 8441 ts_tablet_manager.cc:1918] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:13:31.165347 8441 log.cc:1196] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/7784dc12f60a4638b4763f0bae489078
I20260927 11:13:31.165589 8441 ts_tablet_manager.cc:1939] T 7784dc12f60a4638b4763f0bae489078 P d8f97afedb2f47858a02dcd706afcb75: Deleting consensus metadata
I20260927 11:13:31.165634 8440 ts_tablet_manager.cc:1918] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:13:31.165685 8440 log.cc:1196] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/wal/wals/7784dc12f60a4638b4763f0bae489078
I20260927 11:13:31.165864 8442 ts_tablet_manager.cc:1918] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:13:31.165908 8440 ts_tablet_manager.cc:1939] T 7784dc12f60a4638b4763f0bae489078 P 8210201d68e843cc90b1d2e08323e5c0: Deleting consensus metadata
I20260927 11:13:31.165925 8442 log.cc:1196] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/7784dc12f60a4638b4763f0bae489078
I20260927 11:13:31.166148 8442 ts_tablet_manager.cc:1939] T 7784dc12f60a4638b4763f0bae489078 P 2bea4a6f688147be81ffeef3903e5838: Deleting consensus metadata
I20260927 11:13:31.166216 7890 catalog_manager.cc:4841] TS 8210201d68e843cc90b1d2e08323e5c0 (127.0.98.1:41965): tablet 7784dc12f60a4638b4763f0bae489078 (table test-table [id=de3afa281d4b4f0abd4ec306ad2f81d0]) successfully deleted
I20260927 11:13:31.166409 7891 catalog_manager.cc:4841] TS 2bea4a6f688147be81ffeef3903e5838 (127.0.98.3:36081): tablet 7784dc12f60a4638b4763f0bae489078 (table test-table [id=de3afa281d4b4f0abd4ec306ad2f81d0]) successfully deleted
I20260927 11:13:31.166477 7891 catalog_manager.cc:4841] TS d8f97afedb2f47858a02dcd706afcb75 (127.0.98.2:37525): tablet 7784dc12f60a4638b4763f0bae489078 (table test-table [id=de3afa281d4b4f0abd4ec306ad2f81d0]) successfully deleted
Sep 27 11:13:31 dist-test-slave-9hq9 krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 11:13:31 dist-test-slave-9hq9 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:31 dist-test-slave-9hq9 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 11:13:31.185846 8456 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:31.180137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.0.98.62:34979)
0927 11:13:31.180187 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:31.180189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:31.180286 (+ 97us) server_negotiation.cc:407] Connection header received
0927 11:13:31.180344 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:31.180344 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:31.180352 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:31.180394 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:31.181053 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.181525 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.182645 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.182828 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.182878 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.185803 (+ 2925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36598: BlockingRecv error: recv got EOF from 127.0.0.1:36598 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 27 11:13:31 dist-test-slave-9hq9 krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 11:13:31.193799 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.186843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.0.98.62:34979)
0927 11:13:31.186877 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:31.186879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:31.187074 (+ 195us) server_negotiation.cc:407] Connection header received
0927 11:13:31.187174 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:31.187175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:31.187184 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:31.187270 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:31.187790 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.188234 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.189016 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.189210 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.189275 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.191294 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:31.191296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:31.191296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:31.191303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:31.192676 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:31.193068 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:31.193070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:31.193070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:31.193127 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:31.193321 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:31.193322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:31.193323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:31.193425 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:31.193528 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:31.193616 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 11:13:31.193690 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:13:31.195220 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:13:31.195396 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:31.196588 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 30aac6825be44bd7a6002a20eadba025 (DEFAULT_TABLE table=test-table [id=5f8858d793de46068c628a3f245715a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:31.196605 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 30aac6825be44bd7a6002a20eadba025 (DEFAULT_TABLE table=test-table [id=5f8858d793de46068c628a3f245715a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:31.196635 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 30aac6825be44bd7a6002a20eadba025 (DEFAULT_TABLE table=test-table [id=5f8858d793de46068c628a3f245715a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:31.196704 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30aac6825be44bd7a6002a20eadba025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:31.196704 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30aac6825be44bd7a6002a20eadba025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:31.196740 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30aac6825be44bd7a6002a20eadba025. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:31.197885 8417 tablet_bootstrap.cc:492] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0: Bootstrap starting.
I20260927 11:13:31.197880 8420 tablet_bootstrap.cc:492] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75: Bootstrap starting.
I20260927 11:13:31.197950 8419 tablet_bootstrap.cc:492] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838: Bootstrap starting.
I20260927 11:13:31.198221 8417 tablet_bootstrap.cc:654] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:31.198221 8420 tablet_bootstrap.cc:654] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:31.198352 8419 tablet_bootstrap.cc:654] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:31.198956 8420 tablet_bootstrap.cc:492] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75: No bootstrap required, opened a new log
I20260927 11:13:31.199035 8420 ts_tablet_manager.cc:1397] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:31.199056 8419 tablet_bootstrap.cc:492] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838: No bootstrap required, opened a new log
I20260927 11:13:31.199095 8419 ts_tablet_manager.cc:1397] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:31.199155 8420 raft_consensus.cc:348] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.199239 8420 raft_consensus.cc:374] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:31.199226 8419 raft_consensus.cc:348] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.199266 8420 raft_consensus.cc:729] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f97afedb2f47858a02dcd706afcb75, State: Initialized, Role: FOLLOWER
I20260927 11:13:31.199282 8419 raft_consensus.cc:374] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:31.199311 8419 raft_consensus.cc:729] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bea4a6f688147be81ffeef3903e5838, State: Initialized, Role: FOLLOWER
I20260927 11:13:31.199306 8420 consensus_queue.cc:260] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.199348 8419 consensus_queue.cc:260] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.199398 8420 ts_tablet_manager.cc:1428] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:31.199426 8419 ts_tablet_manager.cc:1428] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:31.200070 8417 tablet_bootstrap.cc:492] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0: No bootstrap required, opened a new log
I20260927 11:13:31.200147 8417 ts_tablet_manager.cc:1397] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 11:13:31.200277 8417 raft_consensus.cc:348] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.200345 8417 raft_consensus.cc:374] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:31.200372 8417 raft_consensus.cc:729] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8210201d68e843cc90b1d2e08323e5c0, State: Initialized, Role: FOLLOWER
I20260927 11:13:31.200415 8417 consensus_queue.cc:260] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.200507 8417 ts_tablet_manager.cc:1428] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:31.320149 8424 raft_consensus.cc:482] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:31.320250 8424 raft_consensus.cc:504] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.320588 8424 leader_election.cc:288] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8210201d68e843cc90b1d2e08323e5c0 (127.0.98.1:41965), d8f97afedb2f47858a02dcd706afcb75 (127.0.98.2:37525)
I20260927 11:13:31.324283 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.320869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56637 (local address 127.0.98.2:37525)
0927 11:13:31.320925 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:31.320927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:31.321162 (+ 235us) server_negotiation.cc:407] Connection header received
0927 11:13:31.321245 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:31.321246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:31.321259 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:31.321300 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:31.321464 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.322170 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.323195 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.323628 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.323696 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.323706 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:31.324165 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 11:13:31.324220 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 11:13:31.324285 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.320733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37525 (local address 127.0.98.3:56637)
0927 11:13:31.321112 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:31.321120 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:31.321185 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:31.321309 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:31.321310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:31.321312 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:31.321389 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.321391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.322271 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.322272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.323103 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.323105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.324011 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.324012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.324078 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.324082 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:31.324150 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 11:13:31.324188 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":41,"thread_start_us":55,"threads_started":1}
I20260927 11:13:31.324326 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.320821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49925 (local address 127.0.98.1:41965)
0927 11:13:31.320888 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:31.320891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:31.321014 (+ 123us) server_negotiation.cc:407] Connection header received
0927 11:13:31.321087 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:31.321088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:31.321098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:31.321158 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:31.321381 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.322148 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.323538 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.323963 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.324044 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.324053 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:31.324225 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 11:13:31.324261 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 11:13:31.324582 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30aac6825be44bd7a6002a20eadba025" candidate_uuid: "2bea4a6f688147be81ffeef3903e5838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8f97afedb2f47858a02dcd706afcb75" is_pre_election: true
I20260927 11:13:31.324683 8201 raft_consensus.cc:2454] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bea4a6f688147be81ffeef3903e5838 in term 0.
I20260927 11:13:31.324856 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.320732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41965 (local address 127.0.98.3:49925)
0927 11:13:31.320922 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:31.320934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:31.321032 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:31.321184 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:31.321185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:31.321189 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:31.321272 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.321274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.322225 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.322226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.323447 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.323450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.324044 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.324045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.324118 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.324140 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 11:13:31.324693 (+ 553us) client_negotiation.cc:240] Negotiation successful
0927 11:13:31.324774 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":39,"thread_start_us":76,"threads_started":1}
I20260927 11:13:31.324858 8275 leader_election.cc:302] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bea4a6f688147be81ffeef3903e5838, d8f97afedb2f47858a02dcd706afcb75; no voters:
I20260927 11:13:31.325017 8424 raft_consensus.cc:2781] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:31.325047 8424 raft_consensus.cc:482] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:31.325068 8424 raft_consensus.cc:3037] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:31.325095 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30aac6825be44bd7a6002a20eadba025" candidate_uuid: "2bea4a6f688147be81ffeef3903e5838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8210201d68e843cc90b1d2e08323e5c0" is_pre_election: true
I20260927 11:13:31.325176 8063 raft_consensus.cc:2454] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2bea4a6f688147be81ffeef3903e5838 in term 0.
I20260927 11:13:31.325783 8424 raft_consensus.cc:504] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.325908 8424 leader_election.cc:288] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [CANDIDATE]: Term 1 election: Requested vote from peers 8210201d68e843cc90b1d2e08323e5c0 (127.0.98.1:41965), d8f97afedb2f47858a02dcd706afcb75 (127.0.98.2:37525)
I20260927 11:13:31.326086 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30aac6825be44bd7a6002a20eadba025" candidate_uuid: "2bea4a6f688147be81ffeef3903e5838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8210201d68e843cc90b1d2e08323e5c0"
I20260927 11:13:31.326164 8063 raft_consensus.cc:3037] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:31.326167 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30aac6825be44bd7a6002a20eadba025" candidate_uuid: "2bea4a6f688147be81ffeef3903e5838" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8f97afedb2f47858a02dcd706afcb75"
I20260927 11:13:31.326232 8201 raft_consensus.cc:3037] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:31.326869 8063 raft_consensus.cc:2454] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bea4a6f688147be81ffeef3903e5838 in term 1.
I20260927 11:13:31.326934 8201 raft_consensus.cc:2454] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bea4a6f688147be81ffeef3903e5838 in term 1.
I20260927 11:13:31.327093 8275 leader_election.cc:302] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bea4a6f688147be81ffeef3903e5838, d8f97afedb2f47858a02dcd706afcb75; no voters:
I20260927 11:13:31.327198 8424 raft_consensus.cc:2781] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:31.327333 8424 raft_consensus.cc:686] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [term 1 LEADER]: Becoming Leader. State: Replica: 2bea4a6f688147be81ffeef3903e5838, State: Running, Role: LEADER
I20260927 11:13:31.327399 8424 consensus_queue.cc:237] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.328100 7905 catalog_manager.cc:5495] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bea4a6f688147be81ffeef3903e5838 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2bea4a6f688147be81ffeef3903e5838" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } health_report { overall_health: HEALTHY } } }
I20260927 11:13:31.345862 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.343525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.0.98.3:36081)
0927 11:13:31.343568 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:31.343571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:31.343578 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:13:31.343648 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:31.343649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:31.343668 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:31.343765 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:31.343884 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.344339 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.345099 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.345298 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.345370 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.345548 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:31.345650 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:31.345731 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:31.345771 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 11:13:31.345787 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:13:31.346779 8063 raft_consensus.cc:1261] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Refusing update from remote peer 2bea4a6f688147be81ffeef3903e5838: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:31.346865 8201 raft_consensus.cc:1261] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Refusing update from remote peer 2bea4a6f688147be81ffeef3903e5838: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:31.347131 8424 consensus_queue.cc:1035] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:31.347275 8491 consensus_queue.cc:1035] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:31.350821 7905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:31.350899 7905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:31.351487 7905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42: Sending DeleteTablet for 3 replicas of tablet 30aac6825be44bd7a6002a20eadba025
I20260927 11:13:31.351737 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 30aac6825be44bd7a6002a20eadba025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38912
I20260927 11:13:31.351770 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 30aac6825be44bd7a6002a20eadba025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44448
I20260927 11:13:31.351821 8441 tablet_replica.cc:330] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75: stopping tablet replica
I20260927 11:13:31.351845 8440 tablet_replica.cc:330] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0: stopping tablet replica
I20260927 11:13:31.351864 8441 raft_consensus.cc:2229] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:31.351895 8440 raft_consensus.cc:2229] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:31.351909 8441 raft_consensus.cc:2258] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:31.351930 8440 raft_consensus.cc:2258] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:31.352094 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 30aac6825be44bd7a6002a20eadba025 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51468
I20260927 11:13:31.352173 8442 tablet_replica.cc:330] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838: stopping tablet replica
I20260927 11:13:31.352221 8442 raft_consensus.cc:2229] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:13:31.352296 8442 raft_consensus.cc:2258] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:31.352594 8442 ts_tablet_manager.cc:1905] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:31.352626 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:13:31.352749 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:31.353968 8183 tablet_service.cc:1467] Processing CreateTablet for tablet ad1dc4cf231c4dde9ef61d010b893fd6 (DEFAULT_TABLE table=test-table [id=78170c7b0ca94284b94f16c076b15ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:31.354101 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1dc4cf231c4dde9ef61d010b893fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:31.354099 8442 ts_tablet_manager.cc:1918] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:31.354152 8442 log.cc:1196] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/30aac6825be44bd7a6002a20eadba025
I20260927 11:13:31.354367 8442 ts_tablet_manager.cc:1939] T 30aac6825be44bd7a6002a20eadba025 P 2bea4a6f688147be81ffeef3903e5838: Deleting consensus metadata
I20260927 11:13:31.354930 8441 ts_tablet_manager.cc:1905] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:31.355316 8420 tablet_bootstrap.cc:492] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75: Bootstrap starting.
I20260927 11:13:31.355618 8420 tablet_bootstrap.cc:654] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:31.356194 8441 ts_tablet_manager.cc:1918] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:31.356251 8441 log.cc:1196] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/30aac6825be44bd7a6002a20eadba025
I20260927 11:13:31.356252 8323 tablet_service.cc:1467] Processing CreateTablet for tablet ad1dc4cf231c4dde9ef61d010b893fd6 (DEFAULT_TABLE table=test-table [id=78170c7b0ca94284b94f16c076b15ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:31.356345 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1dc4cf231c4dde9ef61d010b893fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:31.356472 8441 ts_tablet_manager.cc:1939] T 30aac6825be44bd7a6002a20eadba025 P d8f97afedb2f47858a02dcd706afcb75: Deleting consensus metadata
I20260927 11:13:31.357456 8419 tablet_bootstrap.cc:492] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838: Bootstrap starting.
I20260927 11:13:31.357542 8440 ts_tablet_manager.cc:1905] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:31.357831 8419 tablet_bootstrap.cc:654] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:31.358381 8419 tablet_bootstrap.cc:492] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838: No bootstrap required, opened a new log
I20260927 11:13:31.358417 8419 ts_tablet_manager.cc:1397] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:31.358549 8419 raft_consensus.cc:348] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.358606 8419 raft_consensus.cc:374] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:31.358623 8419 raft_consensus.cc:729] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bea4a6f688147be81ffeef3903e5838, State: Initialized, Role: FOLLOWER
I20260927 11:13:31.358660 8419 consensus_queue.cc:260] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.358729 8419 ts_tablet_manager.cc:1428] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:31.358908 8440 ts_tablet_manager.cc:1918] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:31.358966 8440 log.cc:1196] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/wal/wals/30aac6825be44bd7a6002a20eadba025
I20260927 11:13:31.359167 8440 ts_tablet_manager.cc:1939] T 30aac6825be44bd7a6002a20eadba025 P 8210201d68e843cc90b1d2e08323e5c0: Deleting consensus metadata
I20260927 11:13:31.359290 7891 catalog_manager.cc:4841] TS d8f97afedb2f47858a02dcd706afcb75 (127.0.98.2:37525): tablet 30aac6825be44bd7a6002a20eadba025 (table test-table [id=5f8858d793de46068c628a3f245715a3]) successfully deleted
I20260927 11:13:31.359349 7891 catalog_manager.cc:4841] TS 2bea4a6f688147be81ffeef3903e5838 (127.0.98.3:36081): tablet 30aac6825be44bd7a6002a20eadba025 (table test-table [id=5f8858d793de46068c628a3f245715a3]) successfully deleted
I20260927 11:13:31.359859 7890 catalog_manager.cc:4841] TS 8210201d68e843cc90b1d2e08323e5c0 (127.0.98.1:41965): tablet 30aac6825be44bd7a6002a20eadba025 (table test-table [id=5f8858d793de46068c628a3f245715a3]) successfully deleted
I20260927 11:13:31.360006 8043 tablet_service.cc:1467] Processing CreateTablet for tablet ad1dc4cf231c4dde9ef61d010b893fd6 (DEFAULT_TABLE table=test-table [id=78170c7b0ca94284b94f16c076b15ae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:31.360090 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1dc4cf231c4dde9ef61d010b893fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:31.361166 8417 tablet_bootstrap.cc:492] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0: Bootstrap starting.
I20260927 11:13:31.361533 8417 tablet_bootstrap.cc:654] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:31.362078 8417 tablet_bootstrap.cc:492] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0: No bootstrap required, opened a new log
I20260927 11:13:31.362123 8417 ts_tablet_manager.cc:1397] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 11:13:31.362221 8417 raft_consensus.cc:348] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.362272 8417 raft_consensus.cc:374] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:31.362290 8417 raft_consensus.cc:729] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8210201d68e843cc90b1d2e08323e5c0, State: Initialized, Role: FOLLOWER
I20260927 11:13:31.362324 8417 consensus_queue.cc:260] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.362406 8417 ts_tablet_manager.cc:1428] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:31.363241 8420 tablet_bootstrap.cc:492] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75: No bootstrap required, opened a new log
I20260927 11:13:31.363284 8420 ts_tablet_manager.cc:1397] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260927 11:13:31.363389 8420 raft_consensus.cc:348] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.363446 8420 raft_consensus.cc:374] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:31.363466 8420 raft_consensus.cc:729] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f97afedb2f47858a02dcd706afcb75, State: Initialized, Role: FOLLOWER
I20260927 11:13:31.363509 8420 consensus_queue.cc:260] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.363586 8420 ts_tablet_manager.cc:1428] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:31.587672 8425 raft_consensus.cc:482] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:31.587796 8425 raft_consensus.cc:504] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.587986 8425 leader_election.cc:288] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8210201d68e843cc90b1d2e08323e5c0 (127.0.98.1:41965), 2bea4a6f688147be81ffeef3903e5838 (127.0.98.3:36081)
I20260927 11:13:31.588147 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1dc4cf231c4dde9ef61d010b893fd6" candidate_uuid: "d8f97afedb2f47858a02dcd706afcb75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bea4a6f688147be81ffeef3903e5838" is_pre_election: true
I20260927 11:13:31.588155 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1dc4cf231c4dde9ef61d010b893fd6" candidate_uuid: "d8f97afedb2f47858a02dcd706afcb75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8210201d68e843cc90b1d2e08323e5c0" is_pre_election: true
I20260927 11:13:31.588248 8343 raft_consensus.cc:2454] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f97afedb2f47858a02dcd706afcb75 in term 0.
I20260927 11:13:31.588279 8063 raft_consensus.cc:2454] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f97afedb2f47858a02dcd706afcb75 in term 0.
I20260927 11:13:31.588411 8135 leader_election.cc:302] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bea4a6f688147be81ffeef3903e5838, d8f97afedb2f47858a02dcd706afcb75; no voters:
I20260927 11:13:31.588501 8425 raft_consensus.cc:2781] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:31.588542 8425 raft_consensus.cc:482] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:31.588572 8425 raft_consensus.cc:3037] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:31.589207 8425 raft_consensus.cc:504] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.589345 8425 leader_election.cc:288] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [CANDIDATE]: Term 1 election: Requested vote from peers 8210201d68e843cc90b1d2e08323e5c0 (127.0.98.1:41965), 2bea4a6f688147be81ffeef3903e5838 (127.0.98.3:36081)
I20260927 11:13:31.589519 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1dc4cf231c4dde9ef61d010b893fd6" candidate_uuid: "d8f97afedb2f47858a02dcd706afcb75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bea4a6f688147be81ffeef3903e5838"
I20260927 11:13:31.589519 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1dc4cf231c4dde9ef61d010b893fd6" candidate_uuid: "d8f97afedb2f47858a02dcd706afcb75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8210201d68e843cc90b1d2e08323e5c0"
I20260927 11:13:31.589588 8063 raft_consensus.cc:3037] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:31.589603 8343 raft_consensus.cc:3037] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:31.590222 8343 raft_consensus.cc:2454] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f97afedb2f47858a02dcd706afcb75 in term 1.
I20260927 11:13:31.590339 8063 raft_consensus.cc:2454] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f97afedb2f47858a02dcd706afcb75 in term 1.
I20260927 11:13:31.590438 8135 leader_election.cc:302] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bea4a6f688147be81ffeef3903e5838, d8f97afedb2f47858a02dcd706afcb75; no voters:
I20260927 11:13:31.590529 8425 raft_consensus.cc:2781] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:31.590643 8425 raft_consensus.cc:686] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [term 1 LEADER]: Becoming Leader. State: Replica: d8f97afedb2f47858a02dcd706afcb75, State: Running, Role: LEADER
I20260927 11:13:31.590703 8425 consensus_queue.cc:237] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } }
I20260927 11:13:31.591228 7905 catalog_manager.cc:5495] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8f97afedb2f47858a02dcd706afcb75 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d8f97afedb2f47858a02dcd706afcb75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8f97afedb2f47858a02dcd706afcb75" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 } health_report { overall_health: UNKNOWN } } }
I20260927 11:13:31.649135 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.646953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.0.98.2:37525)
0927 11:13:31.646995 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:31.646997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:31.647007 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:13:31.647031 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:31.647032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:31.647041 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:31.647093 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:31.647188 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.647640 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.648471 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.648661 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.648741 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.648883 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:31.648922 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:31.648979 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:31.649039 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 11:13:31.649058 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:13:31.653190 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.651195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38137 (local address 127.0.98.1:41965)
0927 11:13:31.651250 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:31.651252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:31.651287 (+ 35us) server_negotiation.cc:407] Connection header received
0927 11:13:31.651356 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:31.651357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:31.651365 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:31.651414 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:31.651528 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.651977 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.652646 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.652824 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.652880 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.653014 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:31.653038 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:31.653075 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:31.653131 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 11:13:31.653145 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:13:31.653209 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.651141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41965 (local address 127.0.98.2:38137)
0927 11:13:31.651233 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:31.651242 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:31.651299 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:31.651419 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:31.651419 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:31.651420 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:13:31.651448 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.651449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.652046 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.652046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.652572 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.652573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.652895 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.652896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.652939 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.652950 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:13:31.653096 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:13:31.653097 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 11:13:31.653130 (+ 33us) client_negotiation.cc:240] Negotiation successful
0927 11:13:31.653160 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"thread_start_us":43,"threads_started":1}
I20260927 11:13:31.654928 8343 raft_consensus.cc:1261] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Refusing update from remote peer d8f97afedb2f47858a02dcd706afcb75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:31.654928 8063 raft_consensus.cc:1261] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Refusing update from remote peer d8f97afedb2f47858a02dcd706afcb75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:31.655098 8500 consensus_queue.cc:1035] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bea4a6f688147be81ffeef3903e5838" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:31.655195 8425 consensus_queue.cc:1035] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8210201d68e843cc90b1d2e08323e5c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:31.661082 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.658984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60093 (local address 127.0.98.2:37525)
0927 11:13:31.659017 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:31.659019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:31.659066 (+ 47us) server_negotiation.cc:407] Connection header received
0927 11:13:31.659134 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:31.659135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:31.659143 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:31.659193 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:31.659280 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.659730 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.660460 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.660656 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.660723 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.660872 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:31.660899 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:31.660952 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:31.661010 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 11:13:31.661026 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:13:31.661129 8509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:31.658936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37525 (local address 127.0.98.1:60093)
0927 11:13:31.659017 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:31.659027 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:31.659081 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:31.659197 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:31.659198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:31.659199 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 11:13:31.659236 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.659237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.659807 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.659808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.660362 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:31.660363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:31.660736 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:31.660737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:31.660788 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:31.660792 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 11:13:31.660955 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 11:13:31.660957 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:13:31.661006 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 11:13:31.661049 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"thread_start_us":47,"threads_started":1}
I20260927 11:13:31.666653 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:31.666724 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:31.667272 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1adde35a5d7e42e39e4231f247b87f42: Sending DeleteTablet for 3 replicas of tablet ad1dc4cf231c4dde9ef61d010b893fd6
I20260927 11:13:31.667476 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1dc4cf231c4dde9ef61d010b893fd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44448
I20260927 11:13:31.667505 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1dc4cf231c4dde9ef61d010b893fd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38912
I20260927 11:13:31.667557 8440 tablet_replica.cc:330] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0: stopping tablet replica
I20260927 11:13:31.667585 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1dc4cf231c4dde9ef61d010b893fd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51468
I20260927 11:13:31.667608 8440 raft_consensus.cc:2229] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:31.667647 8440 raft_consensus.cc:2258] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:31.667670 8442 tablet_replica.cc:330] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838: stopping tablet replica
I20260927 11:13:31.667687 8441 tablet_replica.cc:330] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75: stopping tablet replica
I20260927 11:13:31.667726 8442 raft_consensus.cc:2229] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:31.667732 8441 raft_consensus.cc:2229] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:13:31.667763 8442 raft_consensus.cc:2258] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:31.667781 8441 raft_consensus.cc:2258] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:31.667941 8440 ts_tablet_manager.cc:1905] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:31.668111 8441 ts_tablet_manager.cc:1905] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:31.668114 8442 ts_tablet_manager.cc:1905] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:31.669134 8440 ts_tablet_manager.cc:1918] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:13:31.669186 8440 log.cc:1196] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-0/wal/wals/ad1dc4cf231c4dde9ef61d010b893fd6
I20260927 11:13:31.669433 8440 ts_tablet_manager.cc:1939] T ad1dc4cf231c4dde9ef61d010b893fd6 P 8210201d68e843cc90b1d2e08323e5c0: Deleting consensus metadata
I20260927 11:13:31.669432 8441 ts_tablet_manager.cc:1918] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:13:31.669436 8442 ts_tablet_manager.cc:1918] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 11:13:31.669497 8442 log.cc:1196] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/ad1dc4cf231c4dde9ef61d010b893fd6
I20260927 11:13:31.669497 8441 log.cc:1196] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/ad1dc4cf231c4dde9ef61d010b893fd6
I20260927 11:13:31.669708 8441 ts_tablet_manager.cc:1939] T ad1dc4cf231c4dde9ef61d010b893fd6 P d8f97afedb2f47858a02dcd706afcb75: Deleting consensus metadata
I20260927 11:13:31.669708 8442 ts_tablet_manager.cc:1939] T ad1dc4cf231c4dde9ef61d010b893fd6 P 2bea4a6f688147be81ffeef3903e5838: Deleting consensus metadata
I20260927 11:13:31.669734 7890 catalog_manager.cc:4841] TS 8210201d68e843cc90b1d2e08323e5c0 (127.0.98.1:41965): tablet ad1dc4cf231c4dde9ef61d010b893fd6 (table test-table [id=78170c7b0ca94284b94f16c076b15ae7]) successfully deleted
I20260927 11:13:31.670006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 7978
I20260927 11:13:31.670096 7891 catalog_manager.cc:4841] TS d8f97afedb2f47858a02dcd706afcb75 (127.0.98.2:37525): tablet ad1dc4cf231c4dde9ef61d010b893fd6 (table test-table [id=78170c7b0ca94284b94f16c076b15ae7]) successfully deleted
I20260927 11:13:31.670156 7891 catalog_manager.cc:4841] TS 2bea4a6f688147be81ffeef3903e5838 (127.0.98.3:36081): tablet ad1dc4cf231c4dde9ef61d010b893fd6 (table test-table [id=78170c7b0ca94284b94f16c076b15ae7]) successfully deleted
I20260927 11:13:31.676986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 8118
I20260927 11:13:31.682235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 8258
I20260927 11:13:31.687558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 7874
2026-09-27T11:13:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2917 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:13:31 dist-test-slave-9hq9 krb5kdc[8513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:13:31 dist-test-slave-9hq9 krb5kdc[8513](info): set up 2 sockets
Sep 27 11:13:31 dist-test-slave-9hq9 krb5kdc[8513](info): commencing operation
krb5kdc: starting...
W20260927 11:13:32.702510 392 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:13:32 dist-test-slave-9hq9 krb5kdc[8513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:13:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:13:32.809290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37735
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:32.885100 8526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:32.885288 8526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:32.885309 8526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:32.886474 8526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:32.886523 8526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:32.886535 8526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:32.886546 8526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:32.886557 8526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:32.888062 8526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:32.888223 8526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:32.888376 8526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:32.892949 8532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:32.893011 8534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:32.892939 8531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:32.893111 8526 server_base.cc:1034] running on GCE node
I20260927 11:13:32.893366 8526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:32.893626 8526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:32.894785 8526 hybrid_clock.cc:648] HybridClock initialized: now 1790507612894771 us; error 26 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T11:13:32Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1207 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:13:32 dist-test-slave-9hq9 krb5kdc[8539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:13:32 dist-test-slave-9hq9 krb5kdc[8539](info): set up 2 sockets
Sep 27 11:13:32 dist-test-slave-9hq9 krb5kdc[8539](info): commencing operation
krb5kdc: starting...
W20260927 11:13:33.918565 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:13:33 dist-test-slave-9hq9 krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:13:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:13:34.027978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36467
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:34.104233 8552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:34.104417 8552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:34.104439 8552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:34.105564 8552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:34.105604 8552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:34.105618 8552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:34.105674 8552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:34.105696 8552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:34.107188 8552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:34.107360 8552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:34.107573 8552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:34.111984 8557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:34.112049 8552 server_base.cc:1034] running on GCE node
W20260927 11:13:34.112066 8558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:34.112157 8560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:34.112404 8552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:34.112675 8552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:34.113834 8552 hybrid_clock.cc:648] HybridClock initialized: now 1790507614113815 us; error 35 us; skew 500 ppm
Sep 27 11:13:34 dist-test-slave-9hq9 krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:34.120942 8552 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:13:34.121416 8552 webserver.cc:460] Webserver started at http://127.0.98.62:33831/ using document root <none> and password file <none>
I20260927 11:13:34.121634 8552 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:34.121701 8552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:34.121840 8552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:34.122900 8552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "b914b5e4a5fb4d91b2a1e24ad902d529"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "763940cb29f46cab6ee59f0513966d42"
server_key_iv: "abab27dd49634b168ce663eeb2998ba3"
server_key_version: "encryptionkey@0"
I20260927 11:13:34.123250 8552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "b914b5e4a5fb4d91b2a1e24ad902d529"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "763940cb29f46cab6ee59f0513966d42"
server_key_iv: "abab27dd49634b168ce663eeb2998ba3"
server_key_version: "encryptionkey@0"
I20260927 11:13:34.124470 8552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:13:34.125139 8567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:34.125299 8552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:34.125402 8552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "b914b5e4a5fb4d91b2a1e24ad902d529"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "763940cb29f46cab6ee59f0513966d42"
server_key_iv: "abab27dd49634b168ce663eeb2998ba3"
server_key_version: "encryptionkey@0"
I20260927 11:13:34.125469 8552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:34.174520 8552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:34.176277 8552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:34.176424 8552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:34.180914 8552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36467
I20260927 11:13:34.180941 8619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36467 every 8 connection(s)
I20260927 11:13:34.181422 8552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:34.182116 8620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:34.182811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 8552
I20260927 11:13:34.182912 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:34.183097 392 external_mini_cluster.cc:1428] Setting key 5c136ae103de468144cfb52f39bc4768
I20260927 11:13:34.184767 8620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529: Bootstrap starting.
I20260927 11:13:34.185317 8620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:34.185596 8620 log.cc:824] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:34.186301 8620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529: No bootstrap required, opened a new log
I20260927 11:13:34.187973 8620 raft_consensus.cc:348] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36467 } }
I20260927 11:13:34.188115 8620 raft_consensus.cc:374] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:34.188160 8620 raft_consensus.cc:729] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b914b5e4a5fb4d91b2a1e24ad902d529, State: Initialized, Role: FOLLOWER
I20260927 11:13:34.188329 8620 consensus_queue.cc:260] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36467 } }
I20260927 11:13:34.188422 8620 raft_consensus.cc:388] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:34.188465 8620 raft_consensus.cc:482] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:34.188512 8620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:34.188989 8620 raft_consensus.cc:504] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36467 } }
I20260927 11:13:34.189098 8620 leader_election.cc:302] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b914b5e4a5fb4d91b2a1e24ad902d529; no voters:
I20260927 11:13:34.189314 8620 leader_election.cc:288] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:34.189342 8625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 1 FOLLOWER]: Leader election won for term 1
Sep 27 11:13:34 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:34.189519 8625 raft_consensus.cc:686] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 1 LEADER]: Becoming Leader. State: Replica: b914b5e4a5fb4d91b2a1e24ad902d529, State: Running, Role: LEADER
I20260927 11:13:34.189606 8620 sys_catalog.cc:564] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:34.189659 8625 consensus_queue.cc:237] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36467 } }
I20260927 11:13:34.190084 8625 sys_catalog.cc:455] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b914b5e4a5fb4d91b2a1e24ad902d529. Latest consensus state: current_term: 1 leader_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36467 } } }
I20260927 11:13:34.190168 8626 sys_catalog.cc:455] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36467 } } }
I20260927 11:13:34.190238 8625 sys_catalog.cc:458] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:34.190469 8626 sys_catalog.cc:458] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:34.190694 8634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:34.191286 8634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:34.193399 8634 catalog_manager.cc:1347] Generated new cluster ID: 692ac69ec89042b2a057d81ac1684f11
I20260927 11:13:34.193441 8634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:13:34.194453 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:34.183575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.0.98.62:36467)
0927 11:13:34.183742 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:34.183751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:34.183785 (+ 34us) server_negotiation.cc:407] Connection header received
0927 11:13:34.184167 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:34.184173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:34.184216 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:34.184358 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:34.185007 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:34.185966 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:34.186808 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:34.187238 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:34.187357 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:34.189875 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:34.189878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:34.189879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:34.189889 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:34.192706 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:34.193133 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:34.193136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:34.193136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:34.193242 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:34.193515 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:34.193516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:34.193517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:34.193816 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:34.193965 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:34.194107 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 11:13:34.194244 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260927 11:13:34.212144 8634 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:34.212647 8634 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:34.216888 8634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529: Generated new TSK 0
I20260927 11:13:34.217043 8634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:13:34.248894 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36467
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:34.326726 8646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:34.326893 8646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:34.326913 8646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:34.327997 8646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:34.328032 8646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:34.328055 8646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:34.329515 8646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:34.329739 8646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:34.329941 8646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:34.334319 8651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:34.334407 8652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:34.334386 8654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:34.334515 8646 server_base.cc:1034] running on GCE node
I20260927 11:13:34.334733 8646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:34.334990 8646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:34.336150 8646 hybrid_clock.cc:648] HybridClock initialized: now 1790507614336132 us; error 34 us; skew 500 ppm
Sep 27 11:13:34 dist-test-slave-9hq9 krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:34.338644 8646 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:13:34.339087 8646 webserver.cc:460] Webserver started at http://127.0.98.1:33691/ using document root <none> and password file <none>
I20260927 11:13:34.339262 8646 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:34.339320 8646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:34.339459 8646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:34.340688 8646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "71ee44ad541a42ebbee94387010fd8a1"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "db58fbdbef0610f0d93c9ec941876d41"
server_key_iv: "f17c0a19df97e8c88031d990f8bb1b58"
server_key_version: "encryptionkey@0"
I20260927 11:13:34.340966 8646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "71ee44ad541a42ebbee94387010fd8a1"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "db58fbdbef0610f0d93c9ec941876d41"
server_key_iv: "f17c0a19df97e8c88031d990f8bb1b58"
server_key_version: "encryptionkey@0"
I20260927 11:13:34.342168 8646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:13:34.342756 8661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:34.342892 8646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:34.342993 8646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "71ee44ad541a42ebbee94387010fd8a1"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "db58fbdbef0610f0d93c9ec941876d41"
server_key_iv: "f17c0a19df97e8c88031d990f8bb1b58"
server_key_version: "encryptionkey@0"
I20260927 11:13:34.343042 8646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:34.366309 8646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:34.367872 8646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:34.367971 8646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:34.368120 8646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:34.368345 8646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:34.368377 8646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:34.368396 8646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:34.368408 8646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:34.373786 8646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33243
I20260927 11:13:34.373864 8774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33243 every 8 connection(s)
I20260927 11:13:34.374159 8646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:13:34 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:34.382529 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:34.374861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52627 (local address 127.0.98.62:36467)
0927 11:13:34.374908 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:34.374910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:34.375059 (+ 149us) server_negotiation.cc:407] Connection header received
0927 11:13:34.375907 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:34.375908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:34.375916 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:34.375967 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:34.376476 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:34.376959 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:34.377863 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:34.378039 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:34.378144 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:34.380073 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:34.380074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:34.380075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:34.380082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:34.381444 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:34.381768 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:34.381769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:34.381769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:34.381817 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:34.382006 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:34.382007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:34.382007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:34.382154 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:34.382205 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:34.382412 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 11:13:34.382469 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:13:34.382663 8777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:34.374817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:52627)
0927 11:13:34.374959 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:34.374994 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:34.375825 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:34.376014 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:34.376017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:34.376242 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:34.376408 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:34.376411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:34.377031 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:34.377031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:34.377794 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:34.377796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:34.378151 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:34.378152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:34.378248 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:34.378621 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:34.378628 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:34.379939 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:34.381512 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:34.381516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:34.381517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:34.381708 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:34.381873 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:34.381873 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:34.381873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:34.381956 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:34.382209 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:34.382213 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:34.382309 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 11:13:34.382387 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 11:13:34.382492 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260927 11:13:34.383162 8775 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36467
I20260927 11:13:34.383262 8775 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:34.383421 8775 heartbeater.cc:511] Master 127.0.98.62:36467 requested a full tablet report, sending...
I20260927 11:13:34.383551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 8646
I20260927 11:13:34.383656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:13:34.383841 392 external_mini_cluster.cc:1428] Setting key f172d1f1c52c3adaf316b4e36bad476b
I20260927 11:13:34.383930 8584 ts_manager.cc:194] Registered new tserver with Master: 71ee44ad541a42ebbee94387010fd8a1 (127.0.98.1:33243)
I20260927 11:13:34.384872 8584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52627
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:13:34.406690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36467
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:34.484036 8780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:34.484233 8780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:34.484257 8780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:34.485332 8780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:34.485365 8780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:34.485390 8780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:34.486759 8780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:34.486923 8780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:34.487080 8780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:34.491940 8786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:34.491906 8788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:34.492129 8780 server_base.cc:1034] running on GCE node
W20260927 11:13:34.491906 8785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:34.492446 8780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:34.492648 8780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:34.493790 8780 hybrid_clock.cc:648] HybridClock initialized: now 1790507614493780 us; error 28 us; skew 500 ppm
Sep 27 11:13:34 dist-test-slave-9hq9 krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:34.499028 8780 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:13:34.499468 8780 webserver.cc:460] Webserver started at http://127.0.98.2:35685/ using document root <none> and password file <none>
I20260927 11:13:34.499678 8780 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:34.499747 8780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:34.499895 8780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:34.500857 8780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "db2f8a61045b4b2691051326c5684278"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "3d528e78e8ed9680f3aac7396460bdd5"
server_key_iv: "5a8b510c8facef52746cdb7b25f91427"
server_key_version: "encryptionkey@0"
I20260927 11:13:34.501220 8780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "db2f8a61045b4b2691051326c5684278"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "3d528e78e8ed9680f3aac7396460bdd5"
server_key_iv: "5a8b510c8facef52746cdb7b25f91427"
server_key_version: "encryptionkey@0"
I20260927 11:13:34.502532 8780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:13:34.503226 8795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:34.503386 8780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:34.503476 8780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "db2f8a61045b4b2691051326c5684278"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "3d528e78e8ed9680f3aac7396460bdd5"
server_key_iv: "5a8b510c8facef52746cdb7b25f91427"
server_key_version: "encryptionkey@0"
I20260927 11:13:34.503553 8780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:34.520748 8780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:34.522513 8780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:34.522670 8780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:34.522876 8780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:34.523168 8780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:34.523221 8780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:34.523254 8780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:34.523283 8780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:34.529017 8780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41769
I20260927 11:13:34.529070 8908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41769 every 8 connection(s)
I20260927 11:13:34.529340 8780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:13:34.530714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 8780
I20260927 11:13:34.530817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:13:34.530983 392 external_mini_cluster.cc:1428] Setting key 1778a452c2c7bcaad980ed134e4a97ff
Sep 27 11:13:34 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:34.538656 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:34.529983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49349 (local address 127.0.98.62:36467)
0927 11:13:34.530027 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:34.530029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:34.530225 (+ 196us) server_negotiation.cc:407] Connection header received
0927 11:13:34.531139 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:34.531141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:34.531153 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:34.531215 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:34.531913 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:34.532391 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:34.533369 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:34.533567 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:34.533656 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:34.536014 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:34.536015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:34.536015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:34.536020 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:34.537265 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:34.537721 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:34.537722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:34.537722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:34.537768 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:34.538019 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:34.538020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:34.538020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:34.538149 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:34.538200 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:34.538520 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 11:13:34.538573 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:13:34.538894 8911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:34.529938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:49349)
0927 11:13:34.530097 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:34.530142 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:34.531029 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:34.531282 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:34.531285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:34.531606 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:34.531828 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:34.531832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:34.532487 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:34.532488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:34.533271 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:34.533273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:34.533667 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:34.533667 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:34.533793 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:34.534298 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:34.534305 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:34.535911 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:34.537344 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:34.537351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:34.537353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:34.537643 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:34.537829 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:34.537830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:34.537830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:34.537956 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:34.538216 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:34.538223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:34.538379 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 11:13:34.538494 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 11:13:34.538680 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 11:13:34.539350 8909 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36467
I20260927 11:13:34.539433 8909 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:34.539551 8909 heartbeater.cc:511] Master 127.0.98.62:36467 requested a full tablet report, sending...
I20260927 11:13:34.539858 8584 ts_manager.cc:194] Registered new tserver with Master: db2f8a61045b4b2691051326c5684278 (127.0.98.2:41769)
I20260927 11:13:34.540256 8584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:13:34.556654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36467
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:34.636248 8914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:34.636552 8914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:34.636591 8914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:34.638620 8914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:34.638682 8914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:34.638731 8914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:34.640807 8914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:34.641055 8914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:34.641260 8914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:34.646914 8919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:34.646947 8922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:34.647035 8920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:34.647322 8914 server_base.cc:1034] running on GCE node
I20260927 11:13:34.647684 8914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:34.647920 8914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:34.649078 8914 hybrid_clock.cc:648] HybridClock initialized: now 1790507614649059 us; error 31 us; skew 500 ppm
Sep 27 11:13:34 dist-test-slave-9hq9 krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:34.651865 8914 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:13:34.652305 8914 webserver.cc:460] Webserver started at http://127.0.98.3:46593/ using document root <none> and password file <none>
I20260927 11:13:34.652486 8914 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:34.652525 8914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:34.652611 8914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:34.653898 8914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "3fd8c64d95e348b9a9d502722bc9bffb"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "16ba6964c394619a85234f5762b53892"
server_key_iv: "2ec45cdb83c47f55231430709f537951"
server_key_version: "encryptionkey@0"
I20260927 11:13:34.654397 8914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3fd8c64d95e348b9a9d502722bc9bffb"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "16ba6964c394619a85234f5762b53892"
server_key_iv: "2ec45cdb83c47f55231430709f537951"
server_key_version: "encryptionkey@0"
I20260927 11:13:34.656209 8914 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 11:13:34.657032 8929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:34.657212 8914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:34.657295 8914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "3fd8c64d95e348b9a9d502722bc9bffb"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "16ba6964c394619a85234f5762b53892"
server_key_iv: "2ec45cdb83c47f55231430709f537951"
server_key_version: "encryptionkey@0"
I20260927 11:13:34.657372 8914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:34.678383 8914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:34.680634 8914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:34.680759 8914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:34.680927 8914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:34.681156 8914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:34.681185 8914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:34.681205 8914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:34.681217 8914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:34.686885 8914 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41881
I20260927 11:13:34.686939 9042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41881 every 8 connection(s)
I20260927 11:13:34.687234 8914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:13:34.690676 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 8914
I20260927 11:13:34.690807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:13:34.691013 392 external_mini_cluster.cc:1428] Setting key 3c90434ee9be4bb0af09657d489f12b8
Sep 27 11:13:34 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:34.696518 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:34.687940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47621 (local address 127.0.98.62:36467)
0927 11:13:34.687997 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:34.687999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:34.688191 (+ 192us) server_negotiation.cc:407] Connection header received
0927 11:13:34.689284 (+ 1093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:34.689285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:34.689297 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:34.689341 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:34.690002 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:34.690568 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:34.691777 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:34.691976 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:34.692063 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:34.693973 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:34.693974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:34.693975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:34.693982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:34.695212 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:34.695578 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:34.695579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:34.695579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:34.695638 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:34.695870 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:34.695871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:34.695871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:34.695995 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:34.696070 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:34.696355 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 11:13:34.696431 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 11:13:34.696705 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:34.687914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:47621)
0927 11:13:34.688082 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:34.688115 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:34.689171 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:34.689402 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:34.689406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:34.689731 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:34.689910 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:34.689915 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:34.690663 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:34.690664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:34.691670 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:34.691673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:34.692071 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:34.692072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:34.692192 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:34.692589 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:34.692596 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:34.693881 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:34.695300 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:34.695306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:34.695308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:34.695506 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:34.695711 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:34.695712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:34.695712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:34.695805 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:34.696061 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:34.696066 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:34.696207 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 11:13:34.696335 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 11:13:34.696478 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260927 11:13:34.697190 9043 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36467
I20260927 11:13:34.697265 9043 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:34.697438 9043 heartbeater.cc:511] Master 127.0.98.62:36467 requested a full tablet report, sending...
I20260927 11:13:34.697782 8584 ts_manager.cc:194] Registered new tserver with Master: 3fd8c64d95e348b9a9d502722bc9bffb (127.0.98.3:41881)
I20260927 11:13:34.698163 8584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47621
I20260927 11:13:34.702407 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 11:13:34.762771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 8646
I20260927 11:13:34.767221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 8780
I20260927 11:13:34.771329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 8914
I20260927 11:13:34.775399 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 8552
I20260927 11:13:34.780042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36467
--webserver_interface=127.0.98.62
--webserver_port=33831
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:34.856037 9052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:34.856248 9052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:34.856282 9052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:34.857358 9052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:34.857422 9052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:34.857445 9052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:34.857474 9052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:34.857510 9052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:34.858946 9052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:34.859221 9052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:34.859457 9052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:34.864071 9060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:34.864058 9058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:34.864058 9057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:34.864094 9052 server_base.cc:1034] running on GCE node
I20260927 11:13:34.864498 9052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:34.864714 9052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:34.865849 9052 hybrid_clock.cc:648] HybridClock initialized: now 1790507614865831 us; error 25 us; skew 500 ppm
Sep 27 11:13:34 dist-test-slave-9hq9 krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:34.868145 9052 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:13:34.868582 9052 webserver.cc:460] Webserver started at http://127.0.98.62:33831/ using document root <none> and password file <none>
I20260927 11:13:34.868813 9052 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:34.868893 9052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:34.870174 9052 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:34.870761 9067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:34.870919 9052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:34.871009 9052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "b914b5e4a5fb4d91b2a1e24ad902d529"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "763940cb29f46cab6ee59f0513966d42"
server_key_iv: "abab27dd49634b168ce663eeb2998ba3"
server_key_version: "encryptionkey@0"
I20260927 11:13:34.871301 9052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:34.891460 9052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:34.893049 9052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:34.893199 9052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:34.896950 9052 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36467
I20260927 11:13:34.896977 9119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36467 every 8 connection(s)
I20260927 11:13:34.897439 9052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:34.898633 9120 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:13:34.899148 9120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529: Bootstrap starting.
I20260927 11:13:34.904171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 9052
I20260927 11:13:34.904433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33243
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36467
--webserver_port=33691
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:13:34.906877 9120 log.cc:824] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:34.908350 9120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:13:34.908641 9120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529: Bootstrap complete.
I20260927 11:13:34.910843 9120 raft_consensus.cc:348] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36467 } }
I20260927 11:13:34.911043 9120 raft_consensus.cc:729] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b914b5e4a5fb4d91b2a1e24ad902d529, State: Initialized, Role: FOLLOWER
I20260927 11:13:34.911211 9120 consensus_queue.cc:260] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36467 } }
I20260927 11:13:34.911270 9120 raft_consensus.cc:388] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:34.911329 9120 raft_consensus.cc:482] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:34.911376 9120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 1 FOLLOWER]: Advancing to term 2
I20260927 11:13:34.912317 9120 raft_consensus.cc:504] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36467 } }
I20260927 11:13:34.912417 9120 leader_election.cc:302] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b914b5e4a5fb4d91b2a1e24ad902d529; no voters:
I20260927 11:13:34.912617 9120 leader_election.cc:288] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 11:13:34.912654 9124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 11:13:34.912766 9124 raft_consensus.cc:686] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [term 2 LEADER]: Becoming Leader. State: Replica: b914b5e4a5fb4d91b2a1e24ad902d529, State: Running, Role: LEADER
I20260927 11:13:34.912853 9124 consensus_queue.cc:237] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36467 } }
I20260927 11:13:34.912859 9120 sys_catalog.cc:564] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:34.913095 9126 sys_catalog.cc:455] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b914b5e4a5fb4d91b2a1e24ad902d529. Latest consensus state: current_term: 2 leader_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36467 } } }
I20260927 11:13:34.913169 9126 sys_catalog.cc:458] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:34.913395 9125 sys_catalog.cc:455] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b914b5e4a5fb4d91b2a1e24ad902d529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36467 } } }
I20260927 11:13:34.913455 9125 sys_catalog.cc:458] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:34.913489 9128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:34.913945 9128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:34.914091 9128 catalog_manager.cc:1259] Loaded cluster ID: 692ac69ec89042b2a057d81ac1684f11
I20260927 11:13:34.914129 9128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:13:34.915277 9128 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:34.915439 9128 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b914b5e4a5fb4d91b2a1e24ad902d529: Loaded TSK: 0
I20260927 11:13:34.915694 9128 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 11:13:34.982968 9122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:34.983183 9122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:34.983219 9122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:34.984335 9122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:34.984412 9122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:34.984459 9122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:34.985867 9122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33691
--tserver_master_addrs=127.0.98.62:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:34.986145 9122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:34.986367 9122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:34.991101 9146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:34.991127 9147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:34.991266 9122 server_base.cc:1034] running on GCE node
W20260927 11:13:34.991107 9149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:34.991549 9122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:34.991796 9122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:34.992957 9122 hybrid_clock.cc:648] HybridClock initialized: now 1790507614992944 us; error 29 us; skew 500 ppm
Sep 27 11:13:34 dist-test-slave-9hq9 krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507614, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:34.995420 9122 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 11:13:34.995824 9122 webserver.cc:460] Webserver started at http://127.0.98.1:33691/ using document root <none> and password file <none>
I20260927 11:13:34.996052 9122 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:34.996134 9122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:34.997465 9122 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:34.998088 9156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:34.998274 9122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:34.998368 9122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "71ee44ad541a42ebbee94387010fd8a1"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "db58fbdbef0610f0d93c9ec941876d41"
server_key_iv: "f17c0a19df97e8c88031d990f8bb1b58"
server_key_version: "encryptionkey@0"
I20260927 11:13:34.998689 9122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:35.005973 9122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:35.008152 9122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:35.008301 9122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:35.008498 9122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:35.008765 9122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:35.008817 9122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:35.008855 9122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:35.008883 9122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:35.015131 9122 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33243
I20260927 11:13:35.015183 9269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33243 every 8 connection(s)
I20260927 11:13:35.015504 9122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:13:35.018924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 9122
I20260927 11:13:35.019246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41769
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36467
--webserver_port=35685
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:35.028602 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:35.017291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:58709)
0927 11:13:35.017477 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:35.017517 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:35.018122 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:35.022598 (+ 4476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:35.022601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:35.023144 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:35.023321 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.023326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.024417 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.024418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.025208 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.025210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.025815 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.025816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.025938 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.026408 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:35.026416 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:35.028408 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
W20260927 11:13:35.028717 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:35.017399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58709 (local address 127.0.98.62:36467)
0927 11:13:35.022211 (+ 4812us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:35.022220 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:35.022260 (+ 40us) server_negotiation.cc:407] Connection header received
0927 11:13:35.022376 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:35.022381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:35.022420 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:35.022571 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:35.023438 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.024311 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.025308 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.025720 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.025845 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.028527 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58709: BlockingRecv error: recv got EOF from 127.0.98.1:58709 (error 108)
Metrics: {"server-negotiator.queue_time_us":4738,"thread_start_us":65,"threads_started":1}
W20260927 11:13:35.029107 9270 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36467 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36467: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:35.033370 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:35.029320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:42191)
0927 11:13:35.029421 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:35.029430 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:35.029477 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:35.029601 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:35.029602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:35.029723 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:35.029759 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.029760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.030416 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.030417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.030918 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.030919 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.031273 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.031274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.031335 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.031743 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:35.031748 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:35.033297 (+ 1549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 11:13:35.033416 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:35.029375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42191 (local address 127.0.98.62:36467)
0927 11:13:35.029406 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:35.029408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:35.029463 (+ 55us) server_negotiation.cc:407] Connection header received
0927 11:13:35.029531 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:35.029531 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:35.029541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:35.029592 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:35.029835 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.030338 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.031009 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.031193 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.031272 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.033358 (+ 2086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42191: BlockingRecv error: recv got EOF from 127.0.98.1:42191 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:35.037695 9272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:35.033820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:49533)
0927 11:13:35.033891 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:35.033901 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:35.033944 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:35.034088 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:35.034089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:35.034207 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:35.034240 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.034241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.034858 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.034859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.035357 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.035358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.035716 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.035717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.035766 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.036156 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:35.036161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:35.037621 (+ 1460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 11:13:35.037729 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:35.033857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49533 (local address 127.0.98.62:36467)
0927 11:13:35.033889 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:35.033891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:35.033931 (+ 40us) server_negotiation.cc:407] Connection header received
0927 11:13:35.033999 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:35.033999 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:35.034009 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:35.034080 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:35.034325 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.034783 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.035435 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.035636 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.035714 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.037678 (+ 1964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49533: BlockingRecv error: recv got EOF from 127.0.98.1:49533 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 11:13:35.038007 9270 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 11:13:35.105540 9274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:35.105724 9274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:35.105746 9274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:35.106876 9274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:35.106914 9274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:35.106940 9274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:35.108234 9274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35685
--tserver_master_addrs=127.0.98.62:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:35.108392 9274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:35.108541 9274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:35.113253 9282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:35.113220 9280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:35.113328 9274 server_base.cc:1034] running on GCE node
W20260927 11:13:35.113220 9279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:35.113627 9274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:35.113874 9274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:35.115027 9274 hybrid_clock.cc:648] HybridClock initialized: now 1790507615115000 us; error 42 us; skew 500 ppm
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507615, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:35.117249 9274 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 11:13:35.117635 9274 webserver.cc:460] Webserver started at http://127.0.98.2:35685/ using document root <none> and password file <none>
I20260927 11:13:35.117841 9274 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:35.117918 9274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:35.119174 9274 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:35.119704 9289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:35.119853 9274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:35.119943 9274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "db2f8a61045b4b2691051326c5684278"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "3d528e78e8ed9680f3aac7396460bdd5"
server_key_iv: "5a8b510c8facef52746cdb7b25f91427"
server_key_version: "encryptionkey@0"
I20260927 11:13:35.120249 9274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:35.146323 9274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:35.147971 9274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:35.148144 9274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:35.148356 9274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:35.148633 9274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:35.148685 9274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:35.148725 9274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:35.148753 9274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:35.154258 9274 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41769
I20260927 11:13:35.154304 9402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41769 every 8 connection(s)
I20260927 11:13:35.154610 9274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 11:13:35.161355 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:35.155195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44135 (local address 127.0.98.62:36467)
0927 11:13:35.155237 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:35.155240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:35.155416 (+ 176us) server_negotiation.cc:407] Connection header received
0927 11:13:35.155923 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:35.155925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:35.155940 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:35.155992 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:35.156874 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.157378 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.158419 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.158618 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.158700 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.161310 (+ 2610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44135: BlockingRecv error: recv got EOF from 127.0.98.2:44135 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:13:35.161361 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:35.155169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:44135)
0927 11:13:35.155302 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:35.155341 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:35.155813 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:35.156057 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:35.156061 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:35.156604 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:35.156775 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.156779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.157471 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.157472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.158323 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.158325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.158709 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.158710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.158825 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.159247 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:35.159254 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:35.161231 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
W20260927 11:13:35.161638 9403 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36467 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36467: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260927 11:13:35.164345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 9274
I20260927 11:13:35.164631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41881
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36467
--webserver_port=46593
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:35.165526 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:35.161832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:55733)
0927 11:13:35.161935 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:35.161943 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:35.161982 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:35.162133 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:35.162134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:35.162235 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:35.162277 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.162278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.162819 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.162819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.163290 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.163291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.163574 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.163574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.163626 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.164037 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:35.164042 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:35.165486 (+ 1444us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90}
W20260927 11:13:35.165673 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:35.161896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55733 (local address 127.0.98.62:36467)
0927 11:13:35.161943 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:35.161944 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:35.161971 (+ 27us) server_negotiation.cc:407] Connection header received
0927 11:13:35.162021 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:35.162022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:35.162029 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:35.162126 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:35.162331 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.162757 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.163349 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.163517 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.163565 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.165592 (+ 2027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55733: BlockingRecv error: recv got EOF from 127.0.98.2:55733 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:35.170233 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:35.165909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:52077)
0927 11:13:35.165990 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:35.166002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:35.166045 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:35.166211 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:35.166212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:35.166311 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:35.166333 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.166334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.167166 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.167167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.167620 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.167621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.168096 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.168097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.168133 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.168532 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:35.168536 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:35.170194 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 11:13:35.170348 9403 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 11:13:35.170499 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:35.166015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52077 (local address 127.0.98.62:36467)
0927 11:13:35.166102 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:35.166105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:35.166113 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:13:35.166134 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:35.166135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:35.166144 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:35.166204 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:35.166397 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.167080 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.167700 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.168016 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.168082 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.170298 (+ 2216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52077: BlockingRecv error: recv got EOF from 127.0.98.2:52077 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 11:13:35.244903 9406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:35.245087 9406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:35.245110 9406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:35.246253 9406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:35.246291 9406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:35.246317 9406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:35.247723 9406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46593
--tserver_master_addrs=127.0.98.62:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:35.247942 9406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:35.248162 9406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:35.252880 9412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:35.252934 9414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:35.252971 9411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:35.252952 9406 server_base.cc:1034] running on GCE node
I20260927 11:13:35.253271 9406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:35.253443 9406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:35.254601 9406 hybrid_clock.cc:648] HybridClock initialized: now 1790507615254584 us; error 31 us; skew 500 ppm
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507615, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:35.256937 9406 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 11:13:35.257395 9406 webserver.cc:460] Webserver started at http://127.0.98.3:46593/ using document root <none> and password file <none>
I20260927 11:13:35.257622 9406 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:35.257722 9406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:35.259016 9406 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:35.259560 9421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:35.259720 9406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:35.259795 9406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "3fd8c64d95e348b9a9d502722bc9bffb"
format_stamp: "Formatted at 2026-09-27 11:13:34 on dist-test-slave-9hq9"
server_key: "16ba6964c394619a85234f5762b53892"
server_key_iv: "2ec45cdb83c47f55231430709f537951"
server_key_version: "encryptionkey@0"
I20260927 11:13:35.260083 9406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:35.269382 9406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:35.271093 9406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:35.271196 9406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:35.271350 9406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:35.271572 9406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:35.271602 9406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:35.271622 9406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:35.271636 9406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:35.277139 9406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41881
I20260927 11:13:35.277210 9534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41881 every 8 connection(s)
I20260927 11:13:35.277503 9406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:13:35.279515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 9406
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 11:13:35.285557 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:35.278195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56229 (local address 127.0.98.62:36467)
0927 11:13:35.278251 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:35.278253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:35.278501 (+ 248us) server_negotiation.cc:407] Connection header received
0927 11:13:35.278934 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:35.278935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:35.278944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:35.278993 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:35.280076 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.280874 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.282230 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.282404 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.282458 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.285503 (+ 3045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56229: BlockingRecv error: recv got EOF from 127.0.98.3:56229 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 11:13:35.285640 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:35.278199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:56229)
0927 11:13:35.278384 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:35.278415 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:35.278857 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:35.279048 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:35.279051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:35.279757 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:35.279953 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.279957 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.280958 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.280959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.282154 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.282156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.282483 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.282484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.282622 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.283120 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:35.283129 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:35.285445 (+ 2316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
W20260927 11:13:35.285836 9535 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36467 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36467: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260927 11:13:35.287415 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:35.280023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.0.98.62:36467)
0927 11:13:35.280131 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:35.280133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:35.280139 (+ 6us) server_negotiation.cc:407] Connection header received
0927 11:13:35.280162 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:35.280167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:35.280177 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:35.280221 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:35.280901 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.281675 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.282625 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.282821 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.282884 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.284120 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:35.284125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:35.284125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:35.284138 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:35.285981 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:35.286394 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:35.286397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:35.286397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:35.286543 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:35.286740 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:35.286741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:35.286741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:35.286992 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:35.287112 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:35.287201 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 11:13:35.287311 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":67,"threads_started":1}
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:35.290303 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:35.286096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:35945)
0927 11:13:35.286142 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:35.286152 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:35.286185 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:35.286320 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:35.286321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:35.286419 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:35.286455 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.286456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.287255 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.287255 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.287890 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.287892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.288367 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.288367 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.288414 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.288685 (+ 271us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:35.288689 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:35.290264 (+ 1575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 11:13:35.290411 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:35.286127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35945 (local address 127.0.98.62:36467)
0927 11:13:35.286166 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:35.286168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:35.286175 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:13:35.286245 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:35.286246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:35.286255 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:35.286311 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:35.286533 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.287177 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.287998 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.288285 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.288347 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.290352 (+ 2005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35945: BlockingRecv error: recv got EOF from 127.0.98.3:35945 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:35 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:35.294059 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:35.290621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:33455)
0927 11:13:35.290679 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:35.290687 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:35.290731 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:35.290812 (+ 81us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:35.290813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:35.290903 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:35.290923 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.290924 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.291436 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.291437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.291881 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:35.291882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.292161 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.292162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:35.292194 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.292585 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:35.292596 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:35.293989 (+ 1393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 11:13:35.294118 9273 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:35.290650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33455 (local address 127.0.98.62:36467)
0927 11:13:35.290686 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:35.290688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:35.290721 (+ 33us) server_negotiation.cc:407] Connection header received
0927 11:13:35.290765 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:35.290766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:35.290772 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:35.290810 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:35.290959 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.291380 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.291944 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:35.292105 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:35.292149 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:35.294045 (+ 1896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33455: BlockingRecv error: recv got EOF from 127.0.98.3:33455 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260927 11:13:35.294194 9535 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 11:13:36 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:36 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:36.043071 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:36.038535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:39113)
0927 11:13:36.038657 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:36.038668 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:36.038722 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:36.038950 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:36.038951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:36.039106 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:36.039150 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:36.039151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:36.039739 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:36.039740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:36.040463 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:36.040466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:36.040824 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:36.040825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:36.040864 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:36.041274 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:36.041278 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:36.043000 (+ 1722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
W20260927 11:13:36.043182 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:36.038689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39113 (local address 127.0.98.62:36467)
0927 11:13:36.038823 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:36.038825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:36.038834 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:13:36.038863 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:36.038864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:36.038874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:36.038933 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:36.039213 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:36.039680 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:36.040541 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:36.040764 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:36.040817 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:36.043083 (+ 2266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39113: BlockingRecv error: recv got EOF from 127.0.98.1:39113 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
Sep 27 11:13:36 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:36 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:36.175387 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:36.170871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:54355)
0927 11:13:36.171028 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:36.171041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:36.171124 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:36.171271 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:36.171272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:36.171438 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:36.171478 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:36.171480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:36.172145 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:36.172146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:36.172684 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:36.172686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:36.173054 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:36.173055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:36.173112 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:36.173544 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:36.173548 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:36.175328 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":81,"threads_started":1}
W20260927 11:13:36.175472 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:36.170965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54355 (local address 127.0.98.62:36467)
0927 11:13:36.171035 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:36.171037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:36.171106 (+ 69us) server_negotiation.cc:407] Connection header received
0927 11:13:36.171185 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:36.171186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:36.171197 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:36.171267 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:36.171553 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:36.172064 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:36.172767 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:36.172964 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:36.173053 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:36.175411 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54355: BlockingRecv error: recv got EOF from 127.0.98.2:54355 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 11:13:36 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:36 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:36.298868 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:36.294498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:48427)
0927 11:13:36.294627 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:36.294636 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:36.294685 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:36.294818 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:36.294818 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:36.294961 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:36.295004 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:36.295005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:36.295585 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:36.295586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:36.296317 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:36.296320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:36.296622 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:36.296623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:36.296658 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:36.297098 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:36.297103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:36.298810 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
W20260927 11:13:36.298956 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:36.294578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48427 (local address 127.0.98.62:36467)
0927 11:13:36.294636 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:36.294638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:36.294678 (+ 40us) server_negotiation.cc:407] Connection header received
0927 11:13:36.294750 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:36.294750 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:36.294761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:36.294800 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:36.295064 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:36.295512 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:36.296383 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:36.296564 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:36.296614 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:36.298890 (+ 2276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48427: BlockingRecv error: recv got EOF from 127.0.98.3:48427 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 11:13:37 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:37 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:37.047644 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:37.043522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:44653)
0927 11:13:37.043634 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:37.043645 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:37.043697 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:37.043853 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:37.043854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:37.044010 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:37.044057 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:37.044058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:37.044680 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:37.044681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:37.045185 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:37.045187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:37.045487 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:37.045488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:37.045524 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:37.045943 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:37.045947 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:37.047583 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
W20260927 11:13:37.047712 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:37.043593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44653 (local address 127.0.98.62:36467)
0927 11:13:37.043725 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:37.043727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:37.043743 (+ 16us) server_negotiation.cc:407] Connection header received
0927 11:13:37.043764 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:37.043764 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:37.043774 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:37.043834 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:37.044121 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:37.044618 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:37.045249 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:37.045430 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:37.045477 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:37.047663 (+ 2186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44653: BlockingRecv error: recv got EOF from 127.0.98.1:44653 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
Sep 27 11:13:37 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:37 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:37.179807 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:37.175810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:42647)
0927 11:13:37.175933 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:37.175944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:37.176010 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:37.176128 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:37.176128 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:37.176272 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:37.176317 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:37.176318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:37.176892 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:37.176893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:37.177386 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:37.177387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:37.177669 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:37.177669 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:37.177704 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:37.178115 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:37.178119 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:37.179754 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":51,"threads_started":1}
W20260927 11:13:37.179908 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:37.175875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42647 (local address 127.0.98.62:36467)
0927 11:13:37.175943 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:37.175945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:37.176010 (+ 65us) server_negotiation.cc:407] Connection header received
0927 11:13:37.176063 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:37.176063 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:37.176073 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:37.176112 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:37.176373 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:37.176831 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:37.177448 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:37.177613 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:37.177659 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:37.179832 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42647: BlockingRecv error: recv got EOF from 127.0.98.2:42647 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 11:13:37 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:37 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:37.303869 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:37.299290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:34687)
0927 11:13:37.299427 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:37.299440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:37.299504 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:37.299665 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:37.299666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:37.299829 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:37.299881 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:37.299883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:37.300578 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:37.300579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:37.301143 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:37.301145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:37.301508 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:37.301509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:37.301574 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:37.302120 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:37.302125 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:37.303760 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
W20260927 11:13:37.303896 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:37.299400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34687 (local address 127.0.98.62:36467)
0927 11:13:37.299460 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:37.299461 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:37.299486 (+ 25us) server_negotiation.cc:407] Connection header received
0927 11:13:37.299577 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:37.299578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:37.299591 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:37.299660 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:37.299968 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:37.300457 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:37.301229 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:37.301424 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:37.301508 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:37.303833 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34687: BlockingRecv error: recv got EOF from 127.0.98.3:34687 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 11:13:38 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:38 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:38.052776 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:38.048250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:40665)
0927 11:13:38.048392 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:38.048402 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:38.048450 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:38.048617 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:38.048618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:38.048854 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:38.048912 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:38.048916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:38.049616 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:38.049617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:38.050215 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:38.050217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:38.050554 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:38.050554 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:38.050589 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:38.051056 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:38.051061 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:38.052724 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
W20260927 11:13:38.052901 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:38.048354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40665 (local address 127.0.98.62:36467)
0927 11:13:38.048488 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:38.048490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:38.048498 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:13:38.048525 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:38.048526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:38.048535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:38.048590 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:38.048998 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:38.049550 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:38.050304 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:38.050498 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:38.050545 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:38.052805 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40665: BlockingRecv error: recv got EOF from 127.0.98.1:40665 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
Sep 27 11:13:38 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:38 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:38.184528 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:38.180256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:46265)
0927 11:13:38.180406 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:38.180415 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:38.180464 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:38.180584 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:38.180585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:38.180737 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:38.180783 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:38.180784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:38.181382 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:38.181382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:38.181886 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:38.181888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:38.182288 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:38.182289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:38.182344 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:38.182813 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:38.182818 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:38.184468 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
W20260927 11:13:38.184597 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:38.180325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46265 (local address 127.0.98.62:36467)
0927 11:13:38.180390 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:38.180392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:38.180467 (+ 75us) server_negotiation.cc:407] Connection header received
0927 11:13:38.180516 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:38.180516 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:38.180526 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:38.180566 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:38.180839 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:38.181312 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:38.181969 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:38.182202 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:38.182266 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:38.184552 (+ 2286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46265: BlockingRecv error: recv got EOF from 127.0.98.2:46265 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 11:13:38 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:38 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:38.309092 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:38.304313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:52059)
0927 11:13:38.304577 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:38.304587 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:38.304669 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:38.304821 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:38.304822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:38.304979 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:38.305021 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:38.305022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:38.305702 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:38.305703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:38.306371 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:38.306373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:38.306834 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:38.306835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:38.306895 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:38.307413 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:38.307419 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:38.309023 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":182,"threads_started":1}
W20260927 11:13:38.309185 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:38.304500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52059 (local address 127.0.98.62:36467)
0927 11:13:38.304569 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:38.304571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:38.304650 (+ 79us) server_negotiation.cc:407] Connection header received
0927 11:13:38.304739 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:38.304740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:38.304754 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:38.304806 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:38.305098 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:38.305603 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:38.306492 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:38.306755 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:38.306826 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:38.309118 (+ 2292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52059: BlockingRecv error: recv got EOF from 127.0.98.3:52059 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 11:13:39 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:39 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:39.057876 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:39.053313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:49215)
0927 11:13:39.053467 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:39.053500 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:39.053573 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:39.053717 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:39.053718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:39.053860 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:39.053906 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:39.053908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:39.054594 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:39.054595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:39.055140 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:39.055141 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:39.055449 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:39.055450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:39.055486 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:39.055998 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:39.056004 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:39.057824 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":112,"threads_started":1}
W20260927 11:13:39.057982 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:39.053404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49215 (local address 127.0.98.62:36467)
0927 11:13:39.053513 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:39.053515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:39.053577 (+ 62us) server_negotiation.cc:407] Connection header received
0927 11:13:39.053636 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:39.053637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:39.053646 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:39.053696 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:39.053966 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:39.054530 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:39.055212 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:39.055392 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:39.055438 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:39.057907 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49215: BlockingRecv error: recv got EOF from 127.0.98.1:49215 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":45,"threads_started":1}
Sep 27 11:13:39 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:39 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:39.197736 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:39.184987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:48457)
0927 11:13:39.185104 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:39.185114 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:39.185180 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:39.185384 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:39.185385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:39.185603 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:39.185659 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:39.185661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:39.186357 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:39.186358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:39.186868 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:39.186869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:39.187161 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:39.187162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:39.187198 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:39.195465 (+ 8267us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:39.195473 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:39.197673 (+ 2200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":54,"threads_started":1}
W20260927 11:13:39.197865 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:39.185020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48457 (local address 127.0.98.62:36467)
0927 11:13:39.185085 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:39.185087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:39.185182 (+ 95us) server_negotiation.cc:407] Connection header received
0927 11:13:39.185278 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:39.185278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:39.185304 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:39.185365 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:39.185737 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:39.186292 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:39.186932 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:39.187104 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:39.187152 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:39.197788 (+ 10636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48457: BlockingRecv error: recv got EOF from 127.0.98.2:48457 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 11:13:39 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:39 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:39.313725 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:39.309608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:58097)
0927 11:13:39.309740 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:39.309756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:39.309807 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:39.309911 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:39.309912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:39.310065 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:39.310109 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:39.310111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:39.310693 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:39.310693 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:39.311179 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:39.311181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:39.311458 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:39.311459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:39.311494 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:39.311894 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:39.311898 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:39.313671 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
W20260927 11:13:39.313854 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:39.309694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58097 (local address 127.0.98.62:36467)
0927 11:13:39.309761 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:39.309763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:39.309797 (+ 34us) server_negotiation.cc:407] Connection header received
0927 11:13:39.309848 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:39.309848 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:39.309859 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:39.309907 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:39.310165 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:39.310626 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:39.311240 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:39.311400 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:39.311445 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:39.313779 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58097: BlockingRecv error: recv got EOF from 127.0.98.3:58097 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 11:13:40 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:40 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:40.062788 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:40.058331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:51969)
0927 11:13:40.058486 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:40.058497 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:40.058548 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:40.058678 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:40.058678 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:40.058826 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:40.058874 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:40.058875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:40.059519 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:40.059520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:40.060063 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:40.060065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:40.060376 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:40.060377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:40.060414 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:40.060842 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:40.060846 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:40.062648 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
W20260927 11:13:40.062803 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:40.058418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51969 (local address 127.0.98.62:36467)
0927 11:13:40.058527 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:40.058529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:40.058539 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:13:40.058594 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:40.058595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:40.058604 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:40.058660 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:40.058934 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:40.059426 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:40.060133 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:40.060316 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:40.060363 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:40.062741 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51969: BlockingRecv error: recv got EOF from 127.0.98.1:51969 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
Sep 27 11:13:40 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:40 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:40.203429 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:40.198234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:43983)
0927 11:13:40.198355 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:40.198385 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:40.198502 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:40.198667 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:40.198668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:40.198841 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:40.198884 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:40.198886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:40.199716 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:40.199717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:40.200679 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:40.200681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:40.201042 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:40.201043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:40.201112 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:40.201559 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:40.201563 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:40.203373 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
W20260927 11:13:40.203526 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:40.198339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43983 (local address 127.0.98.62:36467)
0927 11:13:40.198379 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:40.198381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:40.198457 (+ 76us) server_negotiation.cc:407] Connection header received
0927 11:13:40.198566 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:40.198567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:40.198577 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:40.198658 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:40.198965 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:40.199629 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:40.200752 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:40.200969 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:40.201026 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:40.203460 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43983: BlockingRecv error: recv got EOF from 127.0.98.2:43983 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 27 11:13:40 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:40 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:40.318192 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:40.314214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:44315)
0927 11:13:40.314329 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:40.314344 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:40.314392 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:40.314541 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:40.314541 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:40.314683 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:40.314726 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:40.314727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:40.315368 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:40.315369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:40.315872 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:40.315873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:40.316172 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:40.316172 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:40.316207 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:40.316588 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:40.316591 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:40.318127 (+ 1536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
W20260927 11:13:40.318238 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:40.314257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44315 (local address 127.0.98.62:36467)
0927 11:13:40.314295 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:40.314297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:40.314380 (+ 83us) server_negotiation.cc:407] Connection header received
0927 11:13:40.314458 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:40.314459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:40.314473 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:40.314534 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:40.314787 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:40.315305 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:40.315931 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:40.316116 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:40.316161 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:40.318194 (+ 2033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44315: BlockingRecv error: recv got EOF from 127.0.98.3:44315 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 27 11:13:41 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:41 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:41.067826 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:41.063295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:60831)
0927 11:13:41.063450 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:41.063462 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:41.063526 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:41.063661 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:41.063661 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:41.063806 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:41.063853 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:41.063855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:41.064474 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:41.064474 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:41.065251 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:41.065253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:41.065559 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:41.065560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:41.065595 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:41.066112 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:41.066121 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:41.067769 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
W20260927 11:13:41.067960 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:41.063388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60831 (local address 127.0.98.62:36467)
0927 11:13:41.063523 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:41.063525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:41.063534 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:13:41.063579 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:41.063579 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:41.063589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:41.063648 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:41.063926 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:41.064410 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:41.065320 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:41.065504 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:41.065550 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:41.067852 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60831: BlockingRecv error: recv got EOF from 127.0.98.1:60831 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
Sep 27 11:13:41 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:41 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:41.208727 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:41.203896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:37249)
0927 11:13:41.204435 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:41.204447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:41.204508 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:41.204751 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:41.204753 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:41.204977 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:41.205035 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:41.205037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:41.205659 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:41.205659 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:41.206244 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:41.206246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:41.206584 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:41.206585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:41.206620 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:41.207020 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:41.207024 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:41.208665 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":57,"threads_started":1}
W20260927 11:13:41.208792 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:41.203944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37249 (local address 127.0.98.62:36467)
0927 11:13:41.204003 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:41.204005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:41.204534 (+ 529us) server_negotiation.cc:407] Connection header received
0927 11:13:41.204615 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:41.204616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:41.204629 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:41.204688 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:41.205126 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:41.205594 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:41.206315 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:41.206509 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:41.206572 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:41.208749 (+ 2177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37249: BlockingRecv error: recv got EOF from 127.0.98.2:37249 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 11:13:41 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:41 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:41.323535 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:41.318684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:35265)
0927 11:13:41.318838 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:41.318849 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:41.318935 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:41.319109 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:41.319110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:41.319282 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:41.319327 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:41.319328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:41.320017 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:41.320018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:41.320548 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:41.320549 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:41.320902 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:41.320903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:41.320965 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:41.321454 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:41.321459 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:41.323443 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
W20260927 11:13:41.323585 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:41.318755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35265 (local address 127.0.98.62:36467)
0927 11:13:41.318824 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:41.318826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:41.318920 (+ 94us) server_negotiation.cc:407] Connection header received
0927 11:13:41.318987 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:41.318988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:41.319001 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:41.319086 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:41.319432 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:41.319920 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:41.320644 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:41.320820 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:41.320900 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:41.323525 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35265: BlockingRecv error: recv got EOF from 127.0.98.3:35265 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 11:13:42 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:42 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:42.072408 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:42.068231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:43071)
0927 11:13:42.068352 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:42.068365 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:42.068405 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:42.068521 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:42.068521 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:42.068659 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:42.068704 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:42.068705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:42.069270 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:42.069270 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:42.069765 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:42.069766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:42.070157 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:42.070158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:42.070197 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:42.070597 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:42.070601 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:42.072319 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
W20260927 11:13:42.072460 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:42.068271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43071 (local address 127.0.98.62:36467)
0927 11:13:42.068382 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:42.068384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:42.068415 (+ 31us) server_negotiation.cc:407] Connection header received
0927 11:13:42.068451 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:42.068452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:42.068461 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:42.068508 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:42.068763 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:42.069214 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:42.069835 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:42.070038 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:42.070156 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:42.072384 (+ 2228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43071: BlockingRecv error: recv got EOF from 127.0.98.1:43071 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
Sep 27 11:13:42 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:42 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:42.213732 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:42.209165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:45943)
0927 11:13:42.209317 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:42.209327 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:42.209404 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:42.209542 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:42.209542 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:42.209710 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:42.209757 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:42.209759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:42.210523 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:42.210524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:42.211038 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:42.211040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:42.211420 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:42.211421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:42.211476 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:42.211862 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:42.211866 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:42.213627 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
W20260927 11:13:42.213745 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:42.209227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45943 (local address 127.0.98.62:36467)
0927 11:13:42.209298 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:42.209300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:42.209386 (+ 86us) server_negotiation.cc:407] Connection header received
0927 11:13:42.209469 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:42.209470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:42.209480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:42.209535 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:42.209832 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:42.210429 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:42.211134 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:42.211336 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:42.211425 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:42.213697 (+ 2272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45943: BlockingRecv error: recv got EOF from 127.0.98.2:45943 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 11:13:42 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:42 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:42.328016 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:42.323984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:59305)
0927 11:13:42.324104 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:42.324116 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:42.324178 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:42.324282 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:42.324282 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:42.324427 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:42.324474 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:42.324476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:42.325068 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:42.325069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:42.325579 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:42.325581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:42.325872 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:42.325873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:42.325917 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:42.326370 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:42.326375 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:42.327962 (+ 1587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
W20260927 11:13:42.328140 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:42.324008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59305 (local address 127.0.98.62:36467)
0927 11:13:42.324068 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:42.324070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:42.324168 (+ 98us) server_negotiation.cc:407] Connection header received
0927 11:13:42.324226 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:42.324226 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:42.324238 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:42.324276 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:42.324531 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:42.325004 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:42.325644 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:42.325813 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:42.325860 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:42.328044 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59305: BlockingRecv error: recv got EOF from 127.0.98.3:59305 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 11:13:43 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:43 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:43.077421 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:43.073029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:40587)
0927 11:13:43.073164 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:43.073174 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:43.073224 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:43.073385 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:43.073386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:43.073537 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:43.073582 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:43.073584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:43.074230 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:43.074231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:43.074775 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:43.074778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:43.075137 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:43.075138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:43.075178 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:43.075585 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:43.075589 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:43.077360 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":54,"threads_started":1}
W20260927 11:13:43.077548 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:43.073142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40587 (local address 127.0.98.62:36467)
0927 11:13:43.073250 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:43.073252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:43.073266 (+ 14us) server_negotiation.cc:407] Connection header received
0927 11:13:43.073288 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:43.073288 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:43.073298 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:43.073376 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:43.073658 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:43.074168 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:43.074894 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:43.075078 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:43.075130 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:43.077439 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40587: BlockingRecv error: recv got EOF from 127.0.98.1:40587 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
Sep 27 11:13:43 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:43 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:43.218937 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:43.214193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:44577)
0927 11:13:43.214333 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:43.214344 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:43.214413 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:43.214553 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:43.214554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:43.214732 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:43.214775 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:43.214777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:43.215574 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:43.215575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:43.216241 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:43.216242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:43.216625 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:43.216626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:43.216677 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:43.217170 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:43.217175 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:43.218874 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
W20260927 11:13:43.219049 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:43.214252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44577 (local address 127.0.98.62:36467)
0927 11:13:43.214314 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:43.214316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:43.214392 (+ 76us) server_negotiation.cc:407] Connection header received
0927 11:13:43.214476 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:43.214477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:43.214488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:43.214546 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:43.214854 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:43.215493 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:43.216330 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:43.216544 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:43.216625 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:43.218957 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44577: BlockingRecv error: recv got EOF from 127.0.98.2:44577 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 11:13:43 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:43 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:43.333010 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:43.328420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:58883)
0927 11:13:43.328572 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:43.328585 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:43.328651 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:43.328805 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:43.328806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:43.328998 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:43.329044 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:43.329046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:43.329721 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:43.329723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:43.330309 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:43.330311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:43.330680 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:43.330681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:43.330741 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:43.331233 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:43.331239 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:43.332934 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
W20260927 11:13:43.333133 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:43.328548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58883 (local address 127.0.98.62:36467)
0927 11:13:43.328595 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:43.328597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:43.328632 (+ 35us) server_negotiation.cc:407] Connection header received
0927 11:13:43.328713 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:43.328714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:43.328727 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:43.328797 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:43.329122 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:43.329627 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:43.330396 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:43.330595 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:43.330678 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:43.333035 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58883: BlockingRecv error: recv got EOF from 127.0.98.3:58883 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 11:13:44 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:44 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:44.082491 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:44.077809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:46549)
0927 11:13:44.077950 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:44.077961 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:44.078027 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:44.078236 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:44.078237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:44.078397 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:44.078445 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:44.078447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:44.079053 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:44.079053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:44.079571 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:44.079572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:44.079922 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:44.079923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:44.080012 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:44.080535 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:44.080541 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:44.082435 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
W20260927 11:13:44.082633 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:44.077873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46549 (local address 127.0.98.62:36467)
0927 11:13:44.077971 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:44.077974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:44.078012 (+ 38us) server_negotiation.cc:407] Connection header received
0927 11:13:44.078142 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:44.078143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:44.078155 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:44.078218 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:44.078506 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:44.078991 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:44.079645 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:44.079841 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:44.079888 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:44.082531 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46549: BlockingRecv error: recv got EOF from 127.0.98.1:46549 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":49,"threads_started":1}
Sep 27 11:13:44 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:44 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:44.228026 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:44.219320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:60651)
0927 11:13:44.219421 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:44.219436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:44.219493 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:44.219614 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:44.219614 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:44.219765 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:44.219810 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:44.219811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:44.220425 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:44.220426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:44.220964 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:44.220965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:44.221261 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:44.221261 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:44.221297 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:44.226243 (+ 4946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:44.226249 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:44.227972 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":48,"threads_started":1}
W20260927 11:13:44.228149 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:44.219371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60651 (local address 127.0.98.62:36467)
0927 11:13:44.219454 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:44.219456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:44.219477 (+ 21us) server_negotiation.cc:407] Connection header received
0927 11:13:44.219542 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:44.219542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:44.219552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:44.219609 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:44.219869 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:44.220364 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:44.221023 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:44.221189 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:44.221235 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:44.228048 (+ 6813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60651: BlockingRecv error: recv got EOF from 127.0.98.2:60651 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 11:13:44 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:44 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:44.337575 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:44.333451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:48019)
0927 11:13:44.333596 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:44.333606 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:44.333665 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:44.333776 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:44.333777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:44.333928 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:44.333974 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:44.333976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:44.334653 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:44.334654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:44.335163 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:44.335164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:44.335452 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:44.335453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:44.335490 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:44.335896 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:44.335900 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:44.337520 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
W20260927 11:13:44.337689 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:44.333515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48019 (local address 127.0.98.62:36467)
0927 11:13:44.333576 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:44.333578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:44.333664 (+ 86us) server_negotiation.cc:407] Connection header received
0927 11:13:44.333706 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:44.333706 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:44.333717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:44.333772 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:44.334026 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:44.334585 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:44.335226 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:44.335393 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:44.335440 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:44.337596 (+ 2156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48019: BlockingRecv error: recv got EOF from 127.0.98.3:48019 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 11:13:45 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:45 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:45.087386 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:45.082970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:47517)
0927 11:13:45.083150 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:45.083162 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:45.083226 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:45.083345 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:45.083345 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:45.083492 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:45.083550 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:45.083551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:45.084158 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:45.084158 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:45.084668 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:45.084669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:45.084975 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:45.084976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:45.085013 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:45.085585 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:45.085590 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:45.087331 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
W20260927 11:13:45.087523 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:45.083048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47517 (local address 127.0.98.62:36467)
0927 11:13:45.083192 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:45.083193 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:45.083204 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:13:45.083272 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:45.083273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:45.083282 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:45.083332 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:45.083610 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:45.084094 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:45.084735 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:45.084913 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:45.084960 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:45.087436 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47517: BlockingRecv error: recv got EOF from 127.0.98.1:47517 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
Sep 27 11:13:45 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:45 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:45.232921 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:45.228402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:52389)
0927 11:13:45.228549 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:45.228559 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:45.228638 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:45.228768 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:45.228768 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:45.228932 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:45.228976 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:45.228978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:45.229632 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:45.229633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:45.230235 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:45.230237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:45.230607 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:45.230607 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:45.230667 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:45.231173 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:45.231178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:45.232816 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
W20260927 11:13:45.232955 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:45.228464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52389 (local address 127.0.98.62:36467)
0927 11:13:45.228534 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:45.228536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:45.228618 (+ 82us) server_negotiation.cc:407] Connection header received
0927 11:13:45.228697 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:45.228698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:45.228710 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:45.228755 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:45.229056 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:45.229553 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:45.230318 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:45.230523 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:45.230606 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:45.232899 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52389: BlockingRecv error: recv got EOF from 127.0.98.2:52389 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 11:13:45 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:45 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:45.342473 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:45.338270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:54721)
0927 11:13:45.338421 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:45.338434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:45.338498 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:45.338625 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:45.338625 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:45.338774 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:45.338819 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:45.338821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:45.339409 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:45.339410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:45.339910 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:45.339912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:45.340213 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:45.340214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:45.340250 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:45.340657 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:45.340662 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:45.342394 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
W20260927 11:13:45.342572 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:45.338402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54721 (local address 127.0.98.62:36467)
0927 11:13:45.338462 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:45.338463 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:45.338483 (+ 20us) server_negotiation.cc:407] Connection header received
0927 11:13:45.338552 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:45.338553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:45.338565 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:45.338603 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:45.338881 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:45.339348 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:45.339976 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:45.340144 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:45.340191 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:45.342490 (+ 2299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54721: BlockingRecv error: recv got EOF from 127.0.98.3:54721 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 11:13:46 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:46 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:46.092295 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:46.087858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:52385)
0927 11:13:46.088050 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:46.088063 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:46.088129 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:46.088305 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:46.088306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:46.088453 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:46.088499 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:46.088500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:46.089110 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:46.089111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:46.089615 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:46.089616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:46.089927 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:46.089928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:46.089966 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:46.090440 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:46.090444 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:46.092238 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":93,"threads_started":1}
W20260927 11:13:46.092394 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:46.088014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52385 (local address 127.0.98.62:36467)
0927 11:13:46.088146 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:46.088148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:46.088155 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:13:46.088213 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:46.088214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:46.088223 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:46.088277 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:46.088559 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:46.089045 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:46.089686 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:46.089867 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:46.089915 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:46.092324 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52385: BlockingRecv error: recv got EOF from 127.0.98.1:52385 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
Sep 27 11:13:46 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:46 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:46.237848 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:46.233383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:47933)
0927 11:13:46.233584 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:46.233596 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:46.233690 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:46.233836 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:46.233837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:46.234006 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:46.234097 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:46.234099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:46.234755 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:46.234757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:46.235326 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:46.235327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:46.235775 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:46.235776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:46.235831 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:46.236225 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:46.236230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:46.237793 (+ 1563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
W20260927 11:13:46.237938 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:46.233466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47933 (local address 127.0.98.62:36467)
0927 11:13:46.233576 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:46.233578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:46.233671 (+ 93us) server_negotiation.cc:407] Connection header received
0927 11:13:46.233756 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:46.233757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:46.233769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:46.233823 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:46.234178 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:46.234673 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:46.235406 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:46.235694 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:46.235773 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:46.237876 (+ 2103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47933: BlockingRecv error: recv got EOF from 127.0.98.2:47933 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 11:13:46 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:46 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:46.347208 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:46.343106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:55535)
0927 11:13:46.343247 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:46.343263 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:46.343316 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:46.343449 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:46.343450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:46.343599 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:46.343645 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:46.343646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:46.344244 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:46.344244 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:46.344753 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:46.344754 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:46.345046 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:46.345047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:46.345083 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:46.345500 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:46.345504 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:46.347153 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
W20260927 11:13:46.347301 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:46.343151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55535 (local address 127.0.98.62:36467)
0927 11:13:46.343212 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:46.343214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:46.343305 (+ 91us) server_negotiation.cc:407] Connection header received
0927 11:13:46.343368 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:46.343368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:46.343378 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:46.343429 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:46.343707 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:46.344182 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:46.344820 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:46.344989 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:46.345035 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:46.347237 (+ 2202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55535: BlockingRecv error: recv got EOF from 127.0.98.3:55535 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 11:13:47 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:47 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:47.098448 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:47.092754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:55997)
0927 11:13:47.092897 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:47.092909 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:47.092986 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:47.093142 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:47.093143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:47.093363 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:47.093417 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:47.093423 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:47.094476 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:47.094477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:47.095287 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:47.095289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:47.095858 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:47.095859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:47.095898 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:47.096425 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:47.096429 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:47.098383 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
W20260927 11:13:47.098590 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:47.092830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55997 (local address 127.0.98.62:36467)
0927 11:13:47.092988 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:47.092990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:47.093025 (+ 35us) server_negotiation.cc:407] Connection header received
0927 11:13:47.093050 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:47.093051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:47.093065 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:47.093126 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:47.093520 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:47.094395 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:47.095369 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:47.095782 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:47.095857 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:47.098496 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55997: BlockingRecv error: recv got EOF from 127.0.98.1:55997 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
Sep 27 11:13:47 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:47 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:47.242422 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:47.238336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:35429)
0927 11:13:47.238458 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:47.238477 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:47.238535 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:47.238648 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:47.238648 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:47.238794 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:47.238838 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:47.238840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:47.239412 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:47.239413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:47.239905 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:47.239906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:47.240225 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:47.240226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:47.240262 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:47.240677 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:47.240682 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:47.242330 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
W20260927 11:13:47.242453 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:47.238390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35429 (local address 127.0.98.62:36467)
0927 11:13:47.238456 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:47.238458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:47.238520 (+ 62us) server_negotiation.cc:407] Connection header received
0927 11:13:47.238577 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:47.238578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:47.238588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:47.238642 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:47.238891 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:47.239347 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:47.239966 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:47.240156 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:47.240214 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:47.242397 (+ 2183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35429: BlockingRecv error: recv got EOF from 127.0.98.2:35429 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 11:13:47 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:47 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:47.351909 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:47.347740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:45419)
0927 11:13:47.347888 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:47.347898 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:47.347950 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:47.348080 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:47.348080 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:47.348224 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:47.348268 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:47.348270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:47.348844 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:47.348844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:47.349354 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:47.349356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:47.349655 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:47.349656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:47.349693 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:47.350127 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:47.350131 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:47.351817 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
W20260927 11:13:47.351989 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:47.347825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45419 (local address 127.0.98.62:36467)
0927 11:13:47.347893 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:47.347895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:47.347944 (+ 49us) server_negotiation.cc:407] Connection header received
0927 11:13:47.347991 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:47.347992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:47.348002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:47.348061 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:47.348332 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:47.348782 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:47.349424 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:47.349588 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:47.349632 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:47.351920 (+ 2288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45419: BlockingRecv error: recv got EOF from 127.0.98.3:45419 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 11:13:48 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:48 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:48.103439 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:48.099061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:39043)
0927 11:13:48.099210 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:48.099221 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:48.099275 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:48.099406 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:48.099407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:48.099551 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:48.099597 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:48.099599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:48.100219 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:48.100219 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:48.100727 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:48.100728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:48.101107 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:48.101108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:48.101147 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:48.101631 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:48.101636 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:48.103382 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
W20260927 11:13:48.103576 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:48.099150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39043 (local address 127.0.98.62:36467)
0927 11:13:48.099283 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:48.099286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:48.099301 (+ 15us) server_negotiation.cc:407] Connection header received
0927 11:13:48.099321 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:48.099322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:48.099331 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:48.099387 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:48.099662 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:48.100155 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:48.100791 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:48.101024 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:48.101107 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:48.103488 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39043: BlockingRecv error: recv got EOF from 127.0.98.1:39043 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":90,"threads_started":1}
Sep 27 11:13:48 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:48 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:48.247373 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:48.242848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:58073)
0927 11:13:48.243028 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:48.243041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:48.243133 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:48.243301 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:48.243302 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:48.243472 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:48.243516 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:48.243518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:48.244173 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:48.244174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:48.244711 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:48.244712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:48.245103 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:48.245104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:48.245169 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:48.245583 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:48.245587 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:48.247310 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":97,"threads_started":1}
W20260927 11:13:48.247478 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:48.242965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58073 (local address 127.0.98.62:36467)
0927 11:13:48.243032 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:48.243035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:48.243113 (+ 78us) server_negotiation.cc:407] Connection header received
0927 11:13:48.243190 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:48.243191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:48.243209 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:48.243280 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:48.243596 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:48.244090 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:48.244798 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:48.244995 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:48.245103 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:48.247407 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58073: BlockingRecv error: recv got EOF from 127.0.98.2:58073 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 11:13:48 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:48 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:48.356825 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:48.352486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:49227)
0927 11:13:48.352596 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:48.352606 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:48.352689 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:48.352812 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:48.352813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:48.352997 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:48.353044 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:48.353046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:48.353724 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:48.353725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:48.354278 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:48.354279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:48.354670 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:48.354671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:48.354727 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:48.355156 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:48.355160 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:48.356766 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
W20260927 11:13:48.356949 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:48.352616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49227 (local address 127.0.98.62:36467)
0927 11:13:48.352687 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:48.352689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:48.352699 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:13:48.352740 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:48.352741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:48.352754 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:48.352797 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:48.353129 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:48.353628 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:48.354367 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:48.354580 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:48.354676 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:48.356860 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49227: BlockingRecv error: recv got EOF from 127.0.98.3:49227 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 11:13:49 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:49 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:49.108417 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:49.103844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:57483)
0927 11:13:49.103984 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:49.104002 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:49.104090 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:49.104262 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:49.104264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:49.104498 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:49.104560 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:49.104563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:49.105210 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:49.105211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:49.105704 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:49.105705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:49.106044 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:49.106044 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:49.106141 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:49.106615 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:49.106620 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:49.108311 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
W20260927 11:13:49.108449 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:49.103891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57483 (local address 127.0.98.62:36467)
0927 11:13:49.104028 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:49.104030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:49.104068 (+ 38us) server_negotiation.cc:407] Connection header received
0927 11:13:49.104161 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:49.104162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:49.104175 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:49.104242 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:49.104643 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:49.105145 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:49.105796 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:49.105982 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:49.106031 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:49.108381 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57483: BlockingRecv error: recv got EOF from 127.0.98.1:57483 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
Sep 27 11:13:49 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:49 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:49.257516 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:49.247929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:52537)
0927 11:13:49.248082 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:49.248095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:49.248167 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:49.248307 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:49.248307 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:49.248479 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:49.248523 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:49.248525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:49.249195 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:49.249196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:49.249720 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:49.249721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:49.250138 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:49.250139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:49.250196 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:49.255786 (+ 5590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:49.255797 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:49.257460 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
W20260927 11:13:49.257652 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:49.248014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52537 (local address 127.0.98.62:36467)
0927 11:13:49.248086 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:49.248089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:49.248161 (+ 72us) server_negotiation.cc:407] Connection header received
0927 11:13:49.248234 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:49.248234 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:49.248246 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:49.248299 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:49.248610 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:49.249110 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:49.249820 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:49.250017 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:49.250138 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:49.257570 (+ 7432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52537: BlockingRecv error: recv got EOF from 127.0.98.2:52537 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 11:13:49 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:49 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:49.362006 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:49.357416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.3:48223)
0927 11:13:49.357558 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:49.357568 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:49.357648 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:49.357813 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:49.357814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:49.357982 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:49.358028 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:49.358029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:49.358752 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:49.358753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:49.359262 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:49.359264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:49.359609 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:49.359610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:49.359646 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:49.360073 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:49.360077 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:49.361939 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
W20260927 11:13:49.362129 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:49.357553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48223 (local address 127.0.98.62:36467)
0927 11:13:49.357601 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:49.357602 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:49.357631 (+ 29us) server_negotiation.cc:407] Connection header received
0927 11:13:49.357710 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:49.357711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:49.357724 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:49.357788 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:49.358176 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:49.358670 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:49.359357 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:49.359539 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:49.359598 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:49.362017 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48223: BlockingRecv error: recv got EOF from 127.0.98.3:48223 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 27 11:13:50 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:50 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:50.113183 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:50.108983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.1:60379)
0927 11:13:50.109129 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:50.109139 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:50.109188 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:50.109317 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:50.109318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:50.109459 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:50.109504 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:50.109506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.110117 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.110117 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:50.110601 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:50.110602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.110897 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.110897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:50.110932 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:50.111327 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:50.111331 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:50.113079 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
W20260927 11:13:50.113225 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:50.109069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60379 (local address 127.0.98.62:36467)
0927 11:13:50.109177 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:50.109183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:50.109191 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:13:50.109230 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:50.109230 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:50.109240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:50.109302 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:50.109560 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.110031 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.110661 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.110839 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.110886 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:50.113155 (+ 2269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60379: BlockingRecv error: recv got EOF from 127.0.98.1:60379 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":64,"threads_started":1}
Sep 27 11:13:50 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 11:13:50 dist-test-slave-9hq9 krb5kdc[8539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 11:13:50.262521 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:50.258095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36467 (local address 127.0.98.2:42969)
0927 11:13:50.258235 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:50.258246 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:50.258316 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:50.258471 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:50.258472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:50.258647 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:50.258689 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:50.258691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.259379 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.259381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:50.259895 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:50.259897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.260274 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.260274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:50.260328 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:50.260831 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:50.260835 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:50.262468 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36467: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
W20260927 11:13:50.262661 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:50.258150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42969 (local address 127.0.98.62:36467)
0927 11:13:50.258204 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:50.258206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:50.258312 (+ 106us) server_negotiation.cc:407] Connection header received
0927 11:13:50.258376 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:50.258377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:50.258390 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:50.258467 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:50.258778 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.259271 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.259977 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.260163 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.260276 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:50.262548 (+ 2272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42969: BlockingRecv error: recv got EOF from 127.0.98.2:42969 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:13:50.284837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 9122
I20260927 11:13:50.289192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 9274
I20260927 11:13:50.293376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 9406
I20260927 11:13:50.297204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 9052
2026-09-27T11:13:50Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17400 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T11:13:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:50Z Disabled control of system clock
I20260927 11:13:50.323776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46525
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 11:13:50.400183 9602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:50.400365 9602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:50.400383 9602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:50.401407 9602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:50.401438 9602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:50.401449 9602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:50.401461 9602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:50.401472 9602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:50.402709 9602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46525
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:50.402872 9602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:50.403102 9602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:50.407910 9607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:50.407897 9608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:50.408022 9602 server_base.cc:1034] running on GCE node
W20260927 11:13:50.408087 9610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:50.408389 9602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:50.408684 9602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:50.409842 9602 hybrid_clock.cc:648] HybridClock initialized: now 1790507630409830 us; error 28 us; skew 500 ppm
I20260927 11:13:50.410989 9602 webserver.cc:460] Webserver started at http://127.0.98.62:44459/ using document root <none> and password file <none>
I20260927 11:13:50.411216 9602 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:50.411285 9602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:50.411428 9602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:50.412396 9602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "aedd1d1bc2ce474398ce7bccc75ba5c1"
format_stamp: "Formatted at 2026-09-27 11:13:50 on dist-test-slave-9hq9"
server_key: "a9aea3d2dd8404bdb1ef1ee8c657e63a"
server_key_iv: "a3da998475fa9fe5ce010a1d8d72fe8a"
server_key_version: "encryptionkey@0"
I20260927 11:13:50.412743 9602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "aedd1d1bc2ce474398ce7bccc75ba5c1"
format_stamp: "Formatted at 2026-09-27 11:13:50 on dist-test-slave-9hq9"
server_key: "a9aea3d2dd8404bdb1ef1ee8c657e63a"
server_key_iv: "a3da998475fa9fe5ce010a1d8d72fe8a"
server_key_version: "encryptionkey@0"
I20260927 11:13:50.414023 9602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:13:50.414737 9616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:50.414901 9602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:50.414995 9602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "aedd1d1bc2ce474398ce7bccc75ba5c1"
format_stamp: "Formatted at 2026-09-27 11:13:50 on dist-test-slave-9hq9"
server_key: "a9aea3d2dd8404bdb1ef1ee8c657e63a"
server_key_iv: "a3da998475fa9fe5ce010a1d8d72fe8a"
server_key_version: "encryptionkey@0"
I20260927 11:13:50.415076 9602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:50.432497 9602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:50.432794 9602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:50.432946 9602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:50.436612 9602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46525
I20260927 11:13:50.436645 9668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46525 every 8 connection(s)
I20260927 11:13:50.436972 9602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:50.437479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 9602
I20260927 11:13:50.437575 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:50.437736 9669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:50.437773 392 external_mini_cluster.cc:1428] Setting key 838489f8f7ae2e979bc534c2ec7dcc10
I20260927 11:13:50.441314 9669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1: Bootstrap starting.
I20260927 11:13:50.441964 9669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:50.442312 9669 log.cc:824] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:50.443166 9669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1: No bootstrap required, opened a new log
I20260927 11:13:50.443405 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:50.438341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.0.98.62:46525)
0927 11:13:50.438545 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:50.438557 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:50.438581 (+ 24us) server_negotiation.cc:407] Connection header received
0927 11:13:50.438992 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:50.438999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:50.439057 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:50.439170 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:50.439429 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.440277 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.441347 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.441826 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.441915 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:50.442685 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:50.442689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:50.442689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:50.442891 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:50.442930 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:50.442945 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:50.442998 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:50.443111 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 11:13:50.443199 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260927 11:13:50.445088 9669 raft_consensus.cc:348] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aedd1d1bc2ce474398ce7bccc75ba5c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46525 } }
I20260927 11:13:50.445241 9669 raft_consensus.cc:374] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:50.445292 9669 raft_consensus.cc:729] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aedd1d1bc2ce474398ce7bccc75ba5c1, State: Initialized, Role: FOLLOWER
I20260927 11:13:50.445410 9669 consensus_queue.cc:260] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aedd1d1bc2ce474398ce7bccc75ba5c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46525 } }
I20260927 11:13:50.445506 9669 raft_consensus.cc:388] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:50.445550 9669 raft_consensus.cc:482] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:50.445598 9669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:50.446159 9669 raft_consensus.cc:504] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aedd1d1bc2ce474398ce7bccc75ba5c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46525 } }
I20260927 11:13:50.446271 9669 leader_election.cc:302] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aedd1d1bc2ce474398ce7bccc75ba5c1; no voters:
I20260927 11:13:50.446419 9669 leader_election.cc:288] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:50.446465 9674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:50.446622 9674 raft_consensus.cc:686] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [term 1 LEADER]: Becoming Leader. State: Replica: aedd1d1bc2ce474398ce7bccc75ba5c1, State: Running, Role: LEADER
I20260927 11:13:50.446674 9669 sys_catalog.cc:564] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:50.446774 9674 consensus_queue.cc:237] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aedd1d1bc2ce474398ce7bccc75ba5c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46525 } }
I20260927 11:13:50.447139 9676 sys_catalog.cc:455] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aedd1d1bc2ce474398ce7bccc75ba5c1. Latest consensus state: current_term: 1 leader_uuid: "aedd1d1bc2ce474398ce7bccc75ba5c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aedd1d1bc2ce474398ce7bccc75ba5c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46525 } } }
I20260927 11:13:50.447268 9676 sys_catalog.cc:458] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:50.447135 9675 sys_catalog.cc:455] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aedd1d1bc2ce474398ce7bccc75ba5c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aedd1d1bc2ce474398ce7bccc75ba5c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46525 } } }
I20260927 11:13:50.447476 9682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:50.447476 9675 sys_catalog.cc:458] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:50.448153 9682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:50.449564 9682 catalog_manager.cc:1347] Generated new cluster ID: 3996c47bbc87407f846a330221c69d1f
I20260927 11:13:50.449615 9682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:13:50.464978 9682 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:50.465428 9682 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:50.469993 9682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aedd1d1bc2ce474398ce7bccc75ba5c1: Generated new TSK 0
I20260927 11:13:50.470167 9682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:13:50.473242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46525
--builtin_ntp_servers=127.0.98.20:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 11:13:50.554169 9693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:50.554382 9693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:50.554414 9693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:50.555483 9693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:50.555545 9693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:50.555586 9693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:50.556831 9693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:50.557070 9693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:50.557339 9693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:50.562738 9698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:50.562767 9701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:50.562767 9699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:50.562945 9693 server_base.cc:1034] running on GCE node
I20260927 11:13:50.563220 9693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:50.563455 9693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:50.564612 9693 hybrid_clock.cc:648] HybridClock initialized: now 1790507630564604 us; error 32 us; skew 500 ppm
I20260927 11:13:50.565825 9693 webserver.cc:460] Webserver started at http://127.0.98.1:38417/ using document root <none> and password file <none>
I20260927 11:13:50.566100 9693 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:50.566157 9693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:50.566303 9693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:50.567272 9693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "14aba48c65144e4d82f2f8d92bf48c7f"
format_stamp: "Formatted at 2026-09-27 11:13:50 on dist-test-slave-9hq9"
server_key: "9423daefc33de9aafe8c92557cadc217"
server_key_iv: "bd0400b8f6d5ec0ff0a6cffe9e131c21"
server_key_version: "encryptionkey@0"
I20260927 11:13:50.567601 9693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "14aba48c65144e4d82f2f8d92bf48c7f"
format_stamp: "Formatted at 2026-09-27 11:13:50 on dist-test-slave-9hq9"
server_key: "9423daefc33de9aafe8c92557cadc217"
server_key_iv: "bd0400b8f6d5ec0ff0a6cffe9e131c21"
server_key_version: "encryptionkey@0"
I20260927 11:13:50.568869 9693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:13:50.569608 9707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:50.569773 9693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:50.569875 9693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "14aba48c65144e4d82f2f8d92bf48c7f"
format_stamp: "Formatted at 2026-09-27 11:13:50 on dist-test-slave-9hq9"
server_key: "9423daefc33de9aafe8c92557cadc217"
server_key_iv: "bd0400b8f6d5ec0ff0a6cffe9e131c21"
server_key_version: "encryptionkey@0"
I20260927 11:13:50.569958 9693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:50.582746 9693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:50.583027 9693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:50.583168 9693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:50.583366 9693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:50.583659 9693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:50.583719 9693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:50.583752 9693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:50.583781 9693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:50.589174 9693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41675
I20260927 11:13:50.589263 9820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41675 every 8 connection(s)
I20260927 11:13:50.589519 9693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:13:50.594280 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:50.590192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42883 (local address 127.0.98.62:46525)
0927 11:13:50.590253 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:50.590255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:50.590464 (+ 209us) server_negotiation.cc:407] Connection header received
0927 11:13:50.590930 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:50.590931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:50.590943 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:50.591006 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:50.591605 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.592073 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.592972 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.593188 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.593275 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:50.593957 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:50.593959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:50.593959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:50.593964 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:50.593977 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:50.593981 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:50.594037 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:50.594157 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 11:13:50.594193 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 11:13:50.594472 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:50.590196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46525 (local address 127.0.98.1:42883)
0927 11:13:50.590356 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:50.590383 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:50.590829 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:50.591067 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:50.591070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:50.591079 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:50.591510 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:50.591514 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.592161 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.592162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:50.592862 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:50.592864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.593284 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.593285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:50.593415 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:50.593824 (+ 409us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:13:50.593824 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:50.593849 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:13:50.593869 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:50.594040 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:50.594044 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:50.594068 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 11:13:50.594157 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 11:13:50.594265 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260927 11:13:50.594934 9821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46525
I20260927 11:13:50.595012 9821 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:50.595134 9821 heartbeater.cc:511] Master 127.0.98.62:46525 requested a full tablet report, sending...
I20260927 11:13:50.595579 9633 ts_manager.cc:194] Registered new tserver with Master: 14aba48c65144e4d82f2f8d92bf48c7f (127.0.98.1:41675)
I20260927 11:13:50.596138 9633 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42883
I20260927 11:13:50.598219 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 9693
I20260927 11:13:50.598289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:13:50.598462 392 external_mini_cluster.cc:1428] Setting key be09f0c5e917c380d4a6b87f5687e83d
I20260927 11:13:50.599247 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46525
--builtin_ntp_servers=127.0.98.20:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 11:13:50.675166 9824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:50.675376 9824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:50.675408 9824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:50.676465 9824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:50.676530 9824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:50.676573 9824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:50.677810 9824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:50.678124 9824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:50.678407 9824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:50.683171 9829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:50.683274 9824 server_base.cc:1034] running on GCE node
W20260927 11:13:50.683214 9832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:50.683224 9830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:50.683575 9824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:50.683820 9824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:50.684976 9824 hybrid_clock.cc:648] HybridClock initialized: now 1790507630684967 us; error 27 us; skew 500 ppm
I20260927 11:13:50.686205 9824 webserver.cc:460] Webserver started at http://127.0.98.2:36835/ using document root <none> and password file <none>
I20260927 11:13:50.686441 9824 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:50.686558 9824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:50.686756 9824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:50.687767 9824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "5b1b1cdb7d2342e08ca0c8448c25939c"
format_stamp: "Formatted at 2026-09-27 11:13:50 on dist-test-slave-9hq9"
server_key: "e13f2a3106b427075e57ae594ed3d635"
server_key_iv: "9c7efcf7b278555e782672a6e2a95928"
server_key_version: "encryptionkey@0"
I20260927 11:13:50.688123 9824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5b1b1cdb7d2342e08ca0c8448c25939c"
format_stamp: "Formatted at 2026-09-27 11:13:50 on dist-test-slave-9hq9"
server_key: "e13f2a3106b427075e57ae594ed3d635"
server_key_iv: "9c7efcf7b278555e782672a6e2a95928"
server_key_version: "encryptionkey@0"
I20260927 11:13:50.689329 9824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:13:50.689982 9838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:50.690261 9824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:50.690362 9824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "5b1b1cdb7d2342e08ca0c8448c25939c"
format_stamp: "Formatted at 2026-09-27 11:13:50 on dist-test-slave-9hq9"
server_key: "e13f2a3106b427075e57ae594ed3d635"
server_key_iv: "9c7efcf7b278555e782672a6e2a95928"
server_key_version: "encryptionkey@0"
I20260927 11:13:50.690446 9824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:50.707413 9824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:50.707661 9824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:50.707800 9824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:50.707993 9824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:50.708285 9824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:50.708338 9824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:50.708375 9824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:50.708401 9824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:50.714016 9824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38571
I20260927 11:13:50.714113 9951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38571 every 8 connection(s)
I20260927 11:13:50.714401 9824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:13:50.719112 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:50.715025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45889 (local address 127.0.98.62:46525)
0927 11:13:50.715084 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:50.715087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:50.715326 (+ 239us) server_negotiation.cc:407] Connection header received
0927 11:13:50.715733 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:50.715733 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:50.715741 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:50.715793 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:50.716358 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.716813 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.717806 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.717978 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.718044 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:50.718920 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:50.718921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:50.718921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:50.718925 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:50.718936 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:50.718940 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:50.718980 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:50.719042 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 11:13:50.719059 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 11:13:50.719291 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:50.715014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46525 (local address 127.0.98.2:45889)
0927 11:13:50.715200 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:50.715243 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:50.715660 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:50.715838 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:50.715841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:50.715849 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:50.716295 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:50.716298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.716885 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.716886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:50.717732 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:50.717735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.718108 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.718109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:50.718247 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:50.718787 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:13:50.718787 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:50.718821 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:13:50.718851 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:50.718980 (+ 129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:50.718982 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:50.718984 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:13:50.719029 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 11:13:50.719094 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":95,"threads_started":1}
I20260927 11:13:50.719758 9952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46525
I20260927 11:13:50.719826 9952 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:50.719945 9952 heartbeater.cc:511] Master 127.0.98.62:46525 requested a full tablet report, sending...
I20260927 11:13:50.720235 9633 ts_manager.cc:194] Registered new tserver with Master: 5b1b1cdb7d2342e08ca0c8448c25939c (127.0.98.2:38571)
I20260927 11:13:50.720536 9633 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45889
I20260927 11:13:50.723541 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 9824
I20260927 11:13:50.723609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:13:50.723742 392 external_mini_cluster.cc:1428] Setting key cb15001b2c9e0d2d747d847364f9fc1f
I20260927 11:13:50.724538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46525
--builtin_ntp_servers=127.0.98.20:42281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 11:13:50.801493 9955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:50.801662 9955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:50.801682 9955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:50.802695 9955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:50.802729 9955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:50.802752 9955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:50.803984 9955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:50.804143 9955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:50.804378 9955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:13:50.808969 9955 server_base.cc:1034] running on GCE node
W20260927 11:13:50.809059 9963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:50.809059 9961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:50.809128 9960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:50.809422 9955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:50.809675 9955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:50.810853 9955 hybrid_clock.cc:648] HybridClock initialized: now 1790507630810841 us; error 23 us; skew 500 ppm
I20260927 11:13:50.811914 9955 webserver.cc:460] Webserver started at http://127.0.98.3:41313/ using document root <none> and password file <none>
I20260927 11:13:50.812125 9955 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:50.812191 9955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:50.812346 9955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:50.813279 9955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "847935d177384f7595a85c81b21dd3f3"
format_stamp: "Formatted at 2026-09-27 11:13:50 on dist-test-slave-9hq9"
server_key: "990ca8ba55c124e83f5f7aecefc22dfb"
server_key_iv: "87383880a669ff6539e6d416ee928a54"
server_key_version: "encryptionkey@0"
I20260927 11:13:50.813611 9955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "847935d177384f7595a85c81b21dd3f3"
format_stamp: "Formatted at 2026-09-27 11:13:50 on dist-test-slave-9hq9"
server_key: "990ca8ba55c124e83f5f7aecefc22dfb"
server_key_iv: "87383880a669ff6539e6d416ee928a54"
server_key_version: "encryptionkey@0"
I20260927 11:13:50.815001 9955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:13:50.815680 9969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:50.815848 9955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:50.815937 9955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "847935d177384f7595a85c81b21dd3f3"
format_stamp: "Formatted at 2026-09-27 11:13:50 on dist-test-slave-9hq9"
server_key: "990ca8ba55c124e83f5f7aecefc22dfb"
server_key_iv: "87383880a669ff6539e6d416ee928a54"
server_key_version: "encryptionkey@0"
I20260927 11:13:50.816020 9955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:50.830106 9955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:50.830351 9955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:50.830499 9955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:50.830696 9955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:50.830966 9955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:50.831017 9955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:50.831048 9955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:50.831071 9955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:50.836468 9955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41915
I20260927 11:13:50.836514 10082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41915 every 8 connection(s)
I20260927 11:13:50.836782 9955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:13:50.838882 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 9955
I20260927 11:13:50.838973 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:13:50.839125 392 external_mini_cluster.cc:1428] Setting key b32682907feb0ec2157550c6c5e807d1
I20260927 11:13:50.841714 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:50.837397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57887 (local address 127.0.98.62:46525)
0927 11:13:50.837476 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:50.837479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:50.837710 (+ 231us) server_negotiation.cc:407] Connection header received
0927 11:13:50.838222 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:50.838224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:50.838237 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:50.838280 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:50.838906 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.839599 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.840658 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.840831 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.840882 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:50.841516 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:50.841517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:50.841518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:50.841521 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:50.841533 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:50.841537 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:50.841575 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:50.841648 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 11:13:50.841665 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 11:13:50.841864 10085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:50.837375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46525 (local address 127.0.98.3:57887)
0927 11:13:50.837575 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:50.837615 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:50.838113 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:50.838350 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:50.838354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:50.838369 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:50.838813 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:50.838817 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.839715 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.839716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:50.840584 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:50.840586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:50.840900 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:50.840900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:50.840994 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:50.841408 (+ 414us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:13:50.841408 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:50.841429 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:13:50.841447 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:50.841585 (+ 138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:50.841589 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:50.841590 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 11:13:50.841635 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 11:13:50.841699 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260927 11:13:50.842312 10083 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46525
I20260927 11:13:50.842422 10083 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:50.842582 10083 heartbeater.cc:511] Master 127.0.98.62:46525 requested a full tablet report, sending...
I20260927 11:13:50.842940 9633 ts_manager.cc:194] Registered new tserver with Master: 847935d177384f7595a85c81b21dd3f3 (127.0.98.3:41915)
I20260927 11:13:50.843261 9633 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57887
I20260927 11:13:50.850538 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 11:13:50.852238 9672 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:50.851733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.0.98.62:46525)
0927 11:13:50.851783 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:50.851785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:50.851917 (+ 132us) server_negotiation.cc:407] Connection header received
0927 11:13:50.851987 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:50.851988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:50.851996 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:50.852064 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:50.852180 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38246: BlockingRecv error: recv got EOF from 127.0.0.1:38246 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:13:50.852349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 9693
I20260927 11:13:50.856664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 9824
I20260927 11:13:50.860771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 9955
I20260927 11:13:50.864590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 9602
2026-09-27T11:13:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (565 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T11:13:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:50Z Disabled control of system clock
I20260927 11:13:50.889253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42787
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 11:13:50.965269 10094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:50.965442 10094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:50.965462 10094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:50.966477 10094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:50.966508 10094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:50.966521 10094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:50.966532 10094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:50.966544 10094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:50.967772 10094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:50.967931 10094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:50.968127 10094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:50.972397 10100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:50.972491 10094 server_base.cc:1034] running on GCE node
W20260927 11:13:50.972422 10099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:50.972364 10102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:50.972795 10094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:50.973078 10094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:50.974251 10094 hybrid_clock.cc:648] HybridClock initialized: now 1790507630974236 us; error 25 us; skew 500 ppm
I20260927 11:13:50.975435 10094 webserver.cc:460] Webserver started at http://127.0.98.62:34875/ using document root <none> and password file <none>
I20260927 11:13:50.975677 10094 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:50.975749 10094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:50.975898 10094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:50.976859 10094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "3bba6978fb994f4588fdc2afc15dae32"
format_stamp: "Formatted at 2026-09-27 11:13:50 on dist-test-slave-9hq9"
server_key: "9fd391082e7837b17a71f28a6d8e1fbb"
server_key_iv: "4e333c592b1e7b5ae31c5c067cb81622"
server_key_version: "encryptionkey@0"
I20260927 11:13:50.977202 10094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "3bba6978fb994f4588fdc2afc15dae32"
format_stamp: "Formatted at 2026-09-27 11:13:50 on dist-test-slave-9hq9"
server_key: "9fd391082e7837b17a71f28a6d8e1fbb"
server_key_iv: "4e333c592b1e7b5ae31c5c067cb81622"
server_key_version: "encryptionkey@0"
I20260927 11:13:50.978446 10094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:13:50.979076 10108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:50.979243 10094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:50.979336 10094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "3bba6978fb994f4588fdc2afc15dae32"
format_stamp: "Formatted at 2026-09-27 11:13:50 on dist-test-slave-9hq9"
server_key: "9fd391082e7837b17a71f28a6d8e1fbb"
server_key_iv: "4e333c592b1e7b5ae31c5c067cb81622"
server_key_version: "encryptionkey@0"
I20260927 11:13:50.979414 10094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:50.982443 10094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:50.982686 10094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:50.982832 10094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:50.986362 10094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42787
I20260927 11:13:50.986399 10160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42787 every 8 connection(s)
I20260927 11:13:50.986701 10094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:50.987272 10161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:50.989669 10161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32: Bootstrap starting.
I20260927 11:13:50.990262 10161 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:50.990535 10161 log.cc:824] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:50.991112 10161 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32: No bootstrap required, opened a new log
I20260927 11:13:50.992594 10161 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bba6978fb994f4588fdc2afc15dae32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } }
I20260927 11:13:50.992738 10161 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:50.992784 10161 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bba6978fb994f4588fdc2afc15dae32, State: Initialized, Role: FOLLOWER
I20260927 11:13:50.992799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 10094
I20260927 11:13:50.992885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:50.992935 10161 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bba6978fb994f4588fdc2afc15dae32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } }
I20260927 11:13:50.993050 10161 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:50.993093 10161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:50.993108 392 external_mini_cluster.cc:1428] Setting key b5f9bb2204521d9b505bd8a047a43591
I20260927 11:13:50.993141 10161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:50.993783 10161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bba6978fb994f4588fdc2afc15dae32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } }
I20260927 11:13:50.993908 10161 leader_election.cc:302] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bba6978fb994f4588fdc2afc15dae32; no voters:
I20260927 11:13:50.994087 10161 leader_election.cc:288] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:50.994256 10167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:50.994346 10161 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:50.994432 10167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [term 1 LEADER]: Becoming Leader. State: Replica: 3bba6978fb994f4588fdc2afc15dae32, State: Running, Role: LEADER
I20260927 11:13:50.994681 10167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bba6978fb994f4588fdc2afc15dae32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } }
I20260927 11:13:50.995146 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:50.993530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.0.98.62:42787)
0927 11:13:50.993737 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:50.993743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:50.993765 (+ 22us) server_negotiation.cc:407] Connection header received
0927 11:13:50.993845 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:50.993850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:50.993879 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:50.994010 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:50.994665 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:50.994668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:50.994668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:50.994672 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:50.994710 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:50.994723 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:50.994810 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:50.994908 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 11:13:50.994968 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":110,"threads_started":1}
I20260927 11:13:50.995124 10167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bba6978fb994f4588fdc2afc15dae32. Latest consensus state: current_term: 1 leader_uuid: "3bba6978fb994f4588fdc2afc15dae32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bba6978fb994f4588fdc2afc15dae32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } }
I20260927 11:13:50.995501 10167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:50.995131 10168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bba6978fb994f4588fdc2afc15dae32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bba6978fb994f4588fdc2afc15dae32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42787 } } }
I20260927 11:13:50.995935 10168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32 [sys.catalog]: This master's current role is: LEADER
W20260927 11:13:50.996402 10182 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:13:50.996457 10182 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:13:50.996544 10177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:50.996749 10177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:50.998014 10177 catalog_manager.cc:1347] Generated new cluster ID: d3dd88aea1de40ca85a4d8d5c89ebc85
I20260927 11:13:50.998072 10177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:13:51.028445 10177 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:51.028585 10177 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:51.033812 10177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bba6978fb994f4588fdc2afc15dae32: Generated new TSK 0
I20260927 11:13:51.033960 10177 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:13:51.039652 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42787
--builtin_ntp_servers=127.0.98.20:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 11:13:51.115664 10185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:51.115823 10185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:51.115842 10185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:51.116856 10185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:51.116887 10185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:51.116909 10185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:51.118132 10185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:51.118341 10185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:51.118538 10185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:51.123073 10191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:51.123191 10185 server_base.cc:1034] running on GCE node
W20260927 11:13:51.123281 10190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:51.123359 10193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:51.123672 10185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:51.123956 10185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:51.125115 10185 hybrid_clock.cc:648] HybridClock initialized: now 1790507631125106 us; error 28 us; skew 500 ppm
I20260927 11:13:51.126250 10185 webserver.cc:460] Webserver started at http://127.0.98.1:33503/ using document root <none> and password file <none>
I20260927 11:13:51.126416 10185 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:51.126452 10185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:51.126533 10185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:51.127408 10185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "9b8206ea1233412a8bb05c30d7b7d603"
format_stamp: "Formatted at 2026-09-27 11:13:51 on dist-test-slave-9hq9"
server_key: "843759752459a1b7e7d042bec6ad865f"
server_key_iv: "19bd9e3038bc6b9eb35851be9f4cf59b"
server_key_version: "encryptionkey@0"
I20260927 11:13:51.127671 10185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9b8206ea1233412a8bb05c30d7b7d603"
format_stamp: "Formatted at 2026-09-27 11:13:51 on dist-test-slave-9hq9"
server_key: "843759752459a1b7e7d042bec6ad865f"
server_key_iv: "19bd9e3038bc6b9eb35851be9f4cf59b"
server_key_version: "encryptionkey@0"
I20260927 11:13:51.128782 10185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:13:51.129377 10199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:51.129539 10185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:51.129630 10185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "9b8206ea1233412a8bb05c30d7b7d603"
format_stamp: "Formatted at 2026-09-27 11:13:51 on dist-test-slave-9hq9"
server_key: "843759752459a1b7e7d042bec6ad865f"
server_key_iv: "19bd9e3038bc6b9eb35851be9f4cf59b"
server_key_version: "encryptionkey@0"
I20260927 11:13:51.129717 10185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:51.132830 10185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:51.133087 10185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:51.133225 10185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:51.133428 10185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:51.133733 10185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:51.133786 10185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:51.133824 10185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:51.133852 10185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:51.139672 10185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32975
I20260927 11:13:51.139704 10312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32975 every 8 connection(s)
I20260927 11:13:51.140055 10185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:13:51.142823 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:51.140684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49085 (local address 127.0.98.62:42787)
0927 11:13:51.140747 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:51.140749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:51.140966 (+ 217us) server_negotiation.cc:407] Connection header received
0927 11:13:51.141416 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:51.141418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:51.141426 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:51.141474 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:51.142555 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:51.142556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:51.142557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:51.142561 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:51.142577 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:51.142581 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:51.142635 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:51.142741 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 11:13:51.142757 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 11:13:51.143030 10315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:51.140682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42787 (local address 127.0.98.1:49085)
0927 11:13:51.140863 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:51.140891 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:51.141314 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:51.141531 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:51.141534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:51.141542 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:51.142409 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:13:51.142409 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:51.142434 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:13:51.142459 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:51.142647 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:51.142651 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:51.142654 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:51.142751 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 11:13:51.142804 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260927 11:13:51.143553 10313 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42787
I20260927 11:13:51.143657 10313 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:51.143738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 10185
I20260927 11:13:51.143814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:13:51.143814 10313 heartbeater.cc:511] Master 127.0.98.62:42787 requested a full tablet report, sending...
I20260927 11:13:51.143998 392 external_mini_cluster.cc:1428] Setting key ae1d735f0e738b9dcdfa6894ec87ac75
I20260927 11:13:51.144312 10125 ts_manager.cc:194] Registered new tserver with Master: 9b8206ea1233412a8bb05c30d7b7d603 (127.0.98.1:32975)
I20260927 11:13:51.144982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42787
--builtin_ntp_servers=127.0.98.20:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 11:13:51.221020 10316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:51.221194 10316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:51.221226 10316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:51.222381 10316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:51.222419 10316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:51.222445 10316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:51.223809 10316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:51.224046 10316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:51.224323 10316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:51.228942 10324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:51.228936 10321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:51.228936 10322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:51.229084 10316 server_base.cc:1034] running on GCE node
I20260927 11:13:51.229324 10316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:51.229547 10316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:51.230698 10316 hybrid_clock.cc:648] HybridClock initialized: now 1790507631230686 us; error 24 us; skew 500 ppm
I20260927 11:13:51.231734 10316 webserver.cc:460] Webserver started at http://127.0.98.2:33243/ using document root <none> and password file <none>
I20260927 11:13:51.232014 10316 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:51.232089 10316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:51.232228 10316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:51.233237 10316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "90906eab8e6c4c35b43974a7ee0afe88"
format_stamp: "Formatted at 2026-09-27 11:13:51 on dist-test-slave-9hq9"
server_key: "d90a80802aa9c33757f2f1287d1aaad4"
server_key_iv: "efd5edebadc46591c990148a9148f776"
server_key_version: "encryptionkey@0"
I20260927 11:13:51.233609 10316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "90906eab8e6c4c35b43974a7ee0afe88"
format_stamp: "Formatted at 2026-09-27 11:13:51 on dist-test-slave-9hq9"
server_key: "d90a80802aa9c33757f2f1287d1aaad4"
server_key_iv: "efd5edebadc46591c990148a9148f776"
server_key_version: "encryptionkey@0"
I20260927 11:13:51.234848 10316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:13:51.235430 10330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:51.235618 10316 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:51.235714 10316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "90906eab8e6c4c35b43974a7ee0afe88"
format_stamp: "Formatted at 2026-09-27 11:13:51 on dist-test-slave-9hq9"
server_key: "d90a80802aa9c33757f2f1287d1aaad4"
server_key_iv: "efd5edebadc46591c990148a9148f776"
server_key_version: "encryptionkey@0"
I20260927 11:13:51.235788 10316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:51.240140 10316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:51.240382 10316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:51.240509 10316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:51.240707 10316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:51.240995 10316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:51.241026 10316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:51.241044 10316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:51.241089 10316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:51.246469 10316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37899
I20260927 11:13:51.246518 10443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37899 every 8 connection(s)
I20260927 11:13:51.246775 10316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:13:51.249080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 10316
I20260927 11:13:51.249222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:13:51.249337 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:51.247396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33775 (local address 127.0.98.62:42787)
0927 11:13:51.247491 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:51.247493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:51.247665 (+ 172us) server_negotiation.cc:407] Connection header received
0927 11:13:51.248017 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:51.248018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:51.248022 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:51.248071 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:51.249026 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:51.249028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:51.249028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:51.249033 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:51.249053 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:51.249058 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:51.249129 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:51.249251 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 11:13:51.249268 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 11:13:51.249401 392 external_mini_cluster.cc:1428] Setting key f320aaaa0083e91d7dd8db02573080fe
I20260927 11:13:51.249557 10446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:51.247357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42787 (local address 127.0.98.2:33775)
0927 11:13:51.247557 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:51.247585 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:51.247939 (+ 354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:51.248114 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:51.248117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:51.248124 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:51.248891 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:13:51.248892 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:51.248914 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:13:51.248938 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:51.249125 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:51.249127 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:51.249129 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:13:51.249224 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 11:13:51.249302 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":110,"threads_started":1}
I20260927 11:13:51.250087 10444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42787
I20260927 11:13:51.250180 10444 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:51.250283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42787
--builtin_ntp_servers=127.0.98.20:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 11:13:51.250314 10444 heartbeater.cc:511] Master 127.0.98.62:42787 requested a full tablet report, sending...
I20260927 11:13:51.250597 10125 ts_manager.cc:194] Registered new tserver with Master: 90906eab8e6c4c35b43974a7ee0afe88 (127.0.98.2:37899)
W20260927 11:13:51.327471 10447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:51.327643 10447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:51.327664 10447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:51.328702 10447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:51.328734 10447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:51.328759 10447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:51.329959 10447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:51.330180 10447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:51.330396 10447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:51.335047 10453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:51.335047 10455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:51.335135 10452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:51.335192 10447 server_base.cc:1034] running on GCE node
I20260927 11:13:51.335472 10447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:51.335711 10447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:51.336870 10447 hybrid_clock.cc:648] HybridClock initialized: now 1790507631336848 us; error 34 us; skew 500 ppm
I20260927 11:13:51.338006 10447 webserver.cc:460] Webserver started at http://127.0.98.3:46023/ using document root <none> and password file <none>
I20260927 11:13:51.338269 10447 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:51.338342 10447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:51.338470 10447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:51.339591 10447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "85e95a7444b4454cb58854c0c4633167"
format_stamp: "Formatted at 2026-09-27 11:13:51 on dist-test-slave-9hq9"
server_key: "930ba9105a6664d4aae11b8283e7db35"
server_key_iv: "faa1d030855445aa76fc6969dfd1db88"
server_key_version: "encryptionkey@0"
I20260927 11:13:51.339947 10447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "85e95a7444b4454cb58854c0c4633167"
format_stamp: "Formatted at 2026-09-27 11:13:51 on dist-test-slave-9hq9"
server_key: "930ba9105a6664d4aae11b8283e7db35"
server_key_iv: "faa1d030855445aa76fc6969dfd1db88"
server_key_version: "encryptionkey@0"
I20260927 11:13:51.341248 10447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:13:51.341894 10461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:51.342073 10447 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:51.342159 10447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "85e95a7444b4454cb58854c0c4633167"
format_stamp: "Formatted at 2026-09-27 11:13:51 on dist-test-slave-9hq9"
server_key: "930ba9105a6664d4aae11b8283e7db35"
server_key_iv: "faa1d030855445aa76fc6969dfd1db88"
server_key_version: "encryptionkey@0"
I20260927 11:13:51.342247 10447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:51.345391 10447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:51.345635 10447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:51.345770 10447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:51.345978 10447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:51.346274 10447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:51.346341 10447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:51.346397 10447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:51.346426 10447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:51.351975 10447 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40611
I20260927 11:13:51.352007 10574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40611 every 8 connection(s)
I20260927 11:13:51.352311 10447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:13:51.354369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 10447
I20260927 11:13:51.354435 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:13:51.354610 392 external_mini_cluster.cc:1428] Setting key b921833a704c4efe80cb31a8a9cdf11f
I20260927 11:13:51.355093 10165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:51.352981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55245 (local address 127.0.98.62:42787)
0927 11:13:51.353050 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:51.353052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:51.353275 (+ 223us) server_negotiation.cc:407] Connection header received
0927 11:13:51.353647 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:51.353648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:51.353652 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:51.353704 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:51.354843 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:51.354845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:51.354845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:13:51.354849 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:51.354865 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:13:51.354871 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:51.354936 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:51.355010 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 11:13:51.355025 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 11:13:51.355291 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:51.352968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42787 (local address 127.0.98.3:55245)
0927 11:13:51.353149 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:51.353191 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:51.353571 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:51.353750 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:51.353753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:51.353760 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:51.354689 (+ 929us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:13:51.354690 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:51.354721 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:13:51.354753 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:51.354939 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:51.354945 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:51.354947 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:13:51.355016 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 11:13:51.355074 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260927 11:13:51.355666 10575 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42787
I20260927 11:13:51.355734 10575 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:51.355877 10575 heartbeater.cc:511] Master 127.0.98.62:42787 requested a full tablet report, sending...
I20260927 11:13:51.356196 10125 ts_manager.cc:194] Registered new tserver with Master: 85e95a7444b4454cb58854c0c4633167 (127.0.98.3:40611)
I20260927 11:13:51.366487 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 11:13:51.368252 10165 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:13:51.367795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.0.98.62:42787)
0927 11:13:51.367852 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:51.367853 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:51.367951 (+ 98us) server_negotiation.cc:407] Connection header received
0927 11:13:51.368011 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:51.368011 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:51.368014 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:51.368062 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:51.368194 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43664: BlockingRecv error: recv got EOF from 127.0.0.1:43664 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 11:13:51.368366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 10185
I20260927 11:13:51.372504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 10316
I20260927 11:13:51.376598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 10447
I20260927 11:13:51.380317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 10094
2026-09-27T11:13:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (515 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:13:51 dist-test-slave-9hq9 krb5kdc[10585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:13:51 dist-test-slave-9hq9 krb5kdc[10585](info): set up 2 sockets
Sep 27 11:13:51 dist-test-slave-9hq9 krb5kdc[10585](info): commencing operation
krb5kdc: starting...
W20260927 11:13:52.394906 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:13:52 dist-test-slave-9hq9 krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:13:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:13:52.504341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45993
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:52.579738 10598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:52.579952 10598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:52.579984 10598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:52.581103 10598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:13:52.581167 10598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:52.581190 10598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:52.581211 10598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:13:52.581239 10598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:13:52.582609 10598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:52.582870 10598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:52.583109 10598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:52.587630 10604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:52.587638 10606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:52.587638 10603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:52.587984 10598 server_base.cc:1034] running on GCE node
I20260927 11:13:52.588197 10598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:52.588457 10598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:52.589608 10598 hybrid_clock.cc:648] HybridClock initialized: now 1790507632589603 us; error 36 us; skew 500 ppm
Sep 27 11:13:52 dist-test-slave-9hq9 krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:52.591825 10598 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:13:52.592238 10598 webserver.cc:460] Webserver started at http://127.0.98.62:44905/ using document root <none> and password file <none>
I20260927 11:13:52.592448 10598 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:52.592511 10598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:52.592656 10598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:52.593636 10598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "0cde68b748174eccbda61287f7eff209"
format_stamp: "Formatted at 2026-09-27 11:13:52 on dist-test-slave-9hq9"
server_key: "76f45ecf24585608f3f4366ab6b08e24"
server_key_iv: "bad7ce530ce5bc0f941c9a06b6220858"
server_key_version: "encryptionkey@0"
I20260927 11:13:52.593964 10598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "0cde68b748174eccbda61287f7eff209"
format_stamp: "Formatted at 2026-09-27 11:13:52 on dist-test-slave-9hq9"
server_key: "76f45ecf24585608f3f4366ab6b08e24"
server_key_iv: "bad7ce530ce5bc0f941c9a06b6220858"
server_key_version: "encryptionkey@0"
I20260927 11:13:52.595212 10598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:13:52.595888 10613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:52.596047 10598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:13:52.596144 10598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "0cde68b748174eccbda61287f7eff209"
format_stamp: "Formatted at 2026-09-27 11:13:52 on dist-test-slave-9hq9"
server_key: "76f45ecf24585608f3f4366ab6b08e24"
server_key_iv: "bad7ce530ce5bc0f941c9a06b6220858"
server_key_version: "encryptionkey@0"
I20260927 11:13:52.596225 10598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:52.611840 10598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:52.613473 10598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:52.613631 10598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:52.617475 10598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45993
I20260927 11:13:52.617511 10665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45993 every 8 connection(s)
I20260927 11:13:52.617825 10598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:13:52.618507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 10598
I20260927 11:13:52.618621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:13:52.618793 392 external_mini_cluster.cc:1428] Setting key 5cde74e50e727c22d9de1c409c9aa40e
I20260927 11:13:52.619009 10666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:52.621965 10666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209: Bootstrap starting.
I20260927 11:13:52.622612 10666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:52.622918 10666 log.cc:824] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:52.623621 10666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209: No bootstrap required, opened a new log
Sep 27 11:13:52 dist-test-slave-9hq9 krb5kdc[10585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:52.625368 10666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cde68b748174eccbda61287f7eff209" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } }
I20260927 11:13:52.625537 10666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:52.625587 10666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cde68b748174eccbda61287f7eff209, State: Initialized, Role: FOLLOWER
I20260927 11:13:52.625721 10666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cde68b748174eccbda61287f7eff209" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } }
I20260927 11:13:52.625842 10666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:13:52.625877 10666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:13:52.625929 10666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:52.626428 10666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cde68b748174eccbda61287f7eff209" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } }
I20260927 11:13:52.626551 10666 leader_election.cc:302] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cde68b748174eccbda61287f7eff209; no voters:
I20260927 11:13:52.626760 10666 leader_election.cc:288] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:13:52.626786 10671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:52.626884 10671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [term 1 LEADER]: Becoming Leader. State: Replica: 0cde68b748174eccbda61287f7eff209, State: Running, Role: LEADER
I20260927 11:13:52.626959 10671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cde68b748174eccbda61287f7eff209" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } }
I20260927 11:13:52.626995 10666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:13:52.627276 10672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cde68b748174eccbda61287f7eff209" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cde68b748174eccbda61287f7eff209" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } } }
I20260927 11:13:52.627363 10672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:52.627518 10673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cde68b748174eccbda61287f7eff209. Latest consensus state: current_term: 1 leader_uuid: "0cde68b748174eccbda61287f7eff209" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cde68b748174eccbda61287f7eff209" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45993 } } }
I20260927 11:13:52.627583 10673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209 [sys.catalog]: This master's current role is: LEADER
I20260927 11:13:52.627697 10676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:13:52.628154 10676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:13:52.629040 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:52.619289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.0.98.62:45993)
0927 11:13:52.619470 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:52.619478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:52.619509 (+ 31us) server_negotiation.cc:407] Connection header received
0927 11:13:52.619965 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:52.619971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:52.620012 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:52.620126 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:52.620855 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:52.621662 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:52.622755 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:52.623218 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:52.623302 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:52.625418 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:52.625422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:52.625422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:52.625432 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:52.627429 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:52.627864 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:52.627867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:52.627867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:52.627969 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:52.628220 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:52.628220 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:52.628221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:52.628498 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:52.628644 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:52.628708 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 11:13:52.628850 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260927 11:13:52.631021 10676 catalog_manager.cc:1347] Generated new cluster ID: a702f7328c5e4d5887bdb2a48b3494b9
I20260927 11:13:52.631067 10676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:13:52.636766 10688 mvcc.cc:205] Tried to move back new op lower bound from 7333919263248457728 to 7333919263240437760. Current Snapshot: MvccSnapshot[applied={T|T < 7333919263248457728 or (T in {7333919263248457728})}]
I20260927 11:13:52.640926 10676 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:13:52.641595 10676 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:13:52.646198 10676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209: Generated new TSK 0
I20260927 11:13:52.646347 10676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:13:52.674459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45993
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:52.753727 10692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:52.754022 10692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:52.754097 10692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:52.755290 10692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:52.755355 10692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:52.755399 10692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:13:52.756803 10692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:52.757055 10692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:52.757287 10692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:13:52.762450 10692 server_base.cc:1034] running on GCE node
W20260927 11:13:52.762420 10697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:52.762428 10698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:52.762420 10700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:52.762791 10692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:52.763080 10692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:52.764236 10692 hybrid_clock.cc:648] HybridClock initialized: now 1790507632764226 us; error 23 us; skew 500 ppm
Sep 27 11:13:52 dist-test-slave-9hq9 krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:52.766655 10692 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:13:52.767092 10692 webserver.cc:460] Webserver started at http://127.0.98.1:34353/ using document root <none> and password file <none>
I20260927 11:13:52.767318 10692 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:52.767381 10692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:52.767524 10692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:52.768498 10692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "c08dd3b140dc4c93bf686b726688c83a"
format_stamp: "Formatted at 2026-09-27 11:13:52 on dist-test-slave-9hq9"
server_key: "0d3eb5741a4db03a996d7e6aa197d63f"
server_key_iv: "50c8429f52c5581620b64fffb8dce459"
server_key_version: "encryptionkey@0"
I20260927 11:13:52.768831 10692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c08dd3b140dc4c93bf686b726688c83a"
format_stamp: "Formatted at 2026-09-27 11:13:52 on dist-test-slave-9hq9"
server_key: "0d3eb5741a4db03a996d7e6aa197d63f"
server_key_iv: "50c8429f52c5581620b64fffb8dce459"
server_key_version: "encryptionkey@0"
I20260927 11:13:52.770130 10692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:13:52.770846 10707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:52.771063 10692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:52.771188 10692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "c08dd3b140dc4c93bf686b726688c83a"
format_stamp: "Formatted at 2026-09-27 11:13:52 on dist-test-slave-9hq9"
server_key: "0d3eb5741a4db03a996d7e6aa197d63f"
server_key_iv: "50c8429f52c5581620b64fffb8dce459"
server_key_version: "encryptionkey@0"
I20260927 11:13:52.771279 10692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:52.798456 10692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:52.800364 10692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:52.800518 10692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:52.800731 10692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:52.801015 10692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:52.801069 10692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:52.801101 10692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:52.801129 10692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:52.807111 10692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33097
I20260927 11:13:52.807142 10820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33097 every 8 connection(s)
I20260927 11:13:52.807432 10692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:13:52.808542 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 10692
I20260927 11:13:52.808630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:13:52.808792 392 external_mini_cluster.cc:1428] Setting key 27149f5e30679a10b34754408bbdfc15
Sep 27 11:13:52 dist-test-slave-9hq9 krb5kdc[10585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 11:13:52.816382 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:52.808065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45993 (local address 127.0.98.1:46983)
0927 11:13:52.808254 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:52.808291 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:52.808795 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:52.809033 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:52.809037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:52.809558 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:52.809730 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:52.809734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:52.810480 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:52.810481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:52.811265 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:52.811267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:52.811643 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:52.811645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:52.811774 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:52.812245 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:52.812253 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:52.813499 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:52.815059 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:52.815065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:52.815067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:52.815274 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:52.815502 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:52.815503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:52.815504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:52.815606 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:52.815865 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:52.815870 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:52.816007 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 11:13:52.816117 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 11:13:52.816244 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260927 11:13:52.816380 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:52.808103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46983 (local address 127.0.98.62:45993)
0927 11:13:52.808177 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:52.808179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:52.808377 (+ 198us) server_negotiation.cc:407] Connection header received
0927 11:13:52.808894 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:52.808896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:52.808908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:52.808963 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:52.809825 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:52.810383 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:52.811365 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:52.811566 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:52.811639 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:52.813588 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:52.813590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:52.813590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:52.813597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:52.814967 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:52.815361 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:52.815362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:52.815362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:52.815418 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:52.815689 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:52.815690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:52.815690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:52.815798 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:52.815873 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:52.816167 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 11:13:52.816255 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 11:13:52.816816 10821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45993
I20260927 11:13:52.816900 10821 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:52.817113 10821 heartbeater.cc:511] Master 127.0.98.62:45993 requested a full tablet report, sending...
I20260927 11:13:52.817575 10630 ts_manager.cc:194] Registered new tserver with Master: c08dd3b140dc4c93bf686b726688c83a (127.0.98.1:33097)
I20260927 11:13:52.818300 10630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:13:52.832664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45993
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:52.908245 10826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:52.908414 10826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:52.908435 10826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:52.909498 10826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:52.909530 10826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:52.909567 10826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:13:52.910970 10826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:52.911144 10826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:52.911293 10826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:52.915918 10834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:52.915988 10832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:52.916077 10826 server_base.cc:1034] running on GCE node
W20260927 11:13:52.916076 10831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:52.916327 10826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:52.916559 10826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:52.917716 10826 hybrid_clock.cc:648] HybridClock initialized: now 1790507632917712 us; error 32 us; skew 500 ppm
Sep 27 11:13:52 dist-test-slave-9hq9 krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:52.920234 10826 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:13:52.920667 10826 webserver.cc:460] Webserver started at http://127.0.98.2:33565/ using document root <none> and password file <none>
I20260927 11:13:52.920878 10826 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:52.920941 10826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:52.921063 10826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:52.922025 10826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "c254c40149d94cdc96ec97319358f20c"
format_stamp: "Formatted at 2026-09-27 11:13:52 on dist-test-slave-9hq9"
server_key: "6a2b96cae626f08d34c38999be07fab6"
server_key_iv: "c04f9e3d041bd2b6362ddead22e7134d"
server_key_version: "encryptionkey@0"
I20260927 11:13:52.922448 10826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c254c40149d94cdc96ec97319358f20c"
format_stamp: "Formatted at 2026-09-27 11:13:52 on dist-test-slave-9hq9"
server_key: "6a2b96cae626f08d34c38999be07fab6"
server_key_iv: "c04f9e3d041bd2b6362ddead22e7134d"
server_key_version: "encryptionkey@0"
I20260927 11:13:52.923695 10826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:13:52.924369 10841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:52.924547 10826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:52.924638 10826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "c254c40149d94cdc96ec97319358f20c"
format_stamp: "Formatted at 2026-09-27 11:13:52 on dist-test-slave-9hq9"
server_key: "6a2b96cae626f08d34c38999be07fab6"
server_key_iv: "c04f9e3d041bd2b6362ddead22e7134d"
server_key_version: "encryptionkey@0"
I20260927 11:13:52.924721 10826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:52.942453 10826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:52.944707 10826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:52.944864 10826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:52.945081 10826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:52.945372 10826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:52.945425 10826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:52.945456 10826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:52.945483 10826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:52.951027 10826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42531
I20260927 11:13:52.951084 10954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42531 every 8 connection(s)
I20260927 11:13:52.951380 10826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:13:52 dist-test-slave-9hq9 krb5kdc[10585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:52.956838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 10826
I20260927 11:13:52.956936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:13:52.957142 392 external_mini_cluster.cc:1428] Setting key 4001bce0cc0cdaa71ee9a3b3942dd09c
I20260927 11:13:52.960273 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:52.952014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45079 (local address 127.0.98.62:45993)
0927 11:13:52.952055 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:52.952056 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:52.952260 (+ 204us) server_negotiation.cc:407] Connection header received
0927 11:13:52.952693 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:52.952693 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:52.952703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:52.952753 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:52.953572 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:52.954105 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:52.955021 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:52.955205 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:52.955254 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:52.957437 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:52.957438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:52.957439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:52.957446 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:52.958923 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:52.959335 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:52.959336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:52.959337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:52.959391 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:52.959660 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:52.959661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:52.959661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:52.959778 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:52.959850 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:52.960112 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 11:13:52.960184 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:13:52.960469 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:52.952000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45993 (local address 127.0.98.2:45079)
0927 11:13:52.952142 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:52.952173 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:52.952612 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:52.952801 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:52.952804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:52.953318 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:52.953493 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:52.953496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:52.954187 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:52.954187 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:52.954948 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:52.954950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:52.955271 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:52.955272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:52.955364 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:52.955866 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:52.955874 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:52.957314 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:52.959025 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:52.959032 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:52.959034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:52.959246 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:52.959472 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:52.959473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:52.959474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:52.959587 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:52.959842 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:52.959847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:52.959982 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 11:13:52.960089 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 11:13:52.960238 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":86,"threads_started":1}
I20260927 11:13:52.960951 10955 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45993
I20260927 11:13:52.961074 10955 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:52.961244 10955 heartbeater.cc:511] Master 127.0.98.62:45993 requested a full tablet report, sending...
I20260927 11:13:52.961629 10630 ts_manager.cc:194] Registered new tserver with Master: c254c40149d94cdc96ec97319358f20c (127.0.98.2:42531)
I20260927 11:13:52.962180 10630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45079
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:13:52.980032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45993
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:13:53.057431 10960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:13:53.057605 10960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:13:53.057626 10960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:13:53.058724 10960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:13:53.058761 10960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:13:53.058786 10960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:13:53.060145 10960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:13:53.060307 10960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:13:53.060549 10960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:13:53.065171 10965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:53.065194 10960 server_base.cc:1034] running on GCE node
W20260927 11:13:53.065307 10966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:13:53.065356 10968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:13:53.065562 10960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:13:53.065783 10960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:13:53.066929 10960 hybrid_clock.cc:648] HybridClock initialized: now 1790507633066920 us; error 42 us; skew 500 ppm
Sep 27 11:13:53 dist-test-slave-9hq9 krb5kdc[10585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:13:53.069157 10960 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:13:53.069599 10960 webserver.cc:460] Webserver started at http://127.0.98.3:41609/ using document root <none> and password file <none>
I20260927 11:13:53.069795 10960 fs_manager.cc:359] Metadata directory not provided
I20260927 11:13:53.069854 10960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:13:53.069977 10960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:13:53.070874 10960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "c9e9a874be3c423a80da9f2bef6d4161"
format_stamp: "Formatted at 2026-09-27 11:13:53 on dist-test-slave-9hq9"
server_key: "744a1e06287715670fe4c817abc35683"
server_key_iv: "6eeb3bc90d0fa2672991eada0ec6329c"
server_key_version: "encryptionkey@0"
I20260927 11:13:53.071213 10960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c9e9a874be3c423a80da9f2bef6d4161"
format_stamp: "Formatted at 2026-09-27 11:13:53 on dist-test-slave-9hq9"
server_key: "744a1e06287715670fe4c817abc35683"
server_key_iv: "6eeb3bc90d0fa2672991eada0ec6329c"
server_key_version: "encryptionkey@0"
I20260927 11:13:53.072376 10960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:13:53.073030 10975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:53.073189 10960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:13:53.073294 10960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "c9e9a874be3c423a80da9f2bef6d4161"
format_stamp: "Formatted at 2026-09-27 11:13:53 on dist-test-slave-9hq9"
server_key: "744a1e06287715670fe4c817abc35683"
server_key_iv: "6eeb3bc90d0fa2672991eada0ec6329c"
server_key_version: "encryptionkey@0"
I20260927 11:13:53.073376 10960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:13:53.081758 10960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:13:53.083362 10960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:13:53.083514 10960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:13:53.083706 10960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:13:53.084003 10960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:13:53.084048 10960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:53.084084 10960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:13:53.084121 10960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:13:53.089375 10960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44163
I20260927 11:13:53.089412 11088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44163 every 8 connection(s)
I20260927 11:13:53.089666 10960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:13:53.094261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 10960
I20260927 11:13:53.094365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:13:53.094558 392 external_mini_cluster.cc:1428] Setting key 5e60342c025d3f4d25cee23d81e97ca9
Sep 27 11:13:53 dist-test-slave-9hq9 krb5kdc[10585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:13:53.097863 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:53.090304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49783 (local address 127.0.98.62:45993)
0927 11:13:53.090343 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:53.090345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:53.090533 (+ 188us) server_negotiation.cc:407] Connection header received
0927 11:13:53.090891 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:53.090892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:53.090900 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:53.090949 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:53.091717 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.092204 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.093093 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.093268 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.093316 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:53.095480 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:53.095481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:53.095481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:53.095488 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:53.096847 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:53.097216 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:53.097217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:53.097217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:53.097260 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:53.097443 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:53.097443 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:53.097444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:53.097531 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:53.097576 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:53.097756 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 11:13:53.097807 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:13:53.097995 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:53.090242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45993 (local address 127.0.98.3:49783)
0927 11:13:53.090404 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:53.090439 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:53.090823 (+ 384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:53.091012 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:53.091014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:53.091493 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:13:53.091645 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:53.091649 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.092274 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.092275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:53.093026 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:53.093028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.093332 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.093333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:53.093415 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:53.093784 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:13:53.093791 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:13:53.095376 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:13:53.096926 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:53.096936 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:53.096938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:53.097150 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:53.097313 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:13:53.097314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:13:53.097314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:13:53.097394 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:13:53.097581 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:13:53.097585 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:13:53.097675 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 11:13:53.097746 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:13:53.097840 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260927 11:13:53.098519 11089 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45993
I20260927 11:13:53.098620 11089 heartbeater.cc:464] Registering TS with master...
I20260927 11:13:53.098794 11089 heartbeater.cc:511] Master 127.0.98.62:45993 requested a full tablet report, sending...
I20260927 11:13:53.099160 10630 ts_manager.cc:194] Registered new tserver with Master: c9e9a874be3c423a80da9f2bef6d4161 (127.0.98.3:44163)
I20260927 11:13:53.099556 10630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49783
I20260927 11:13:53.106477 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:13:53.113600 10669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:53.107699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.0.98.62:45993)
0927 11:13:53.107766 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:53.107768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:53.107869 (+ 101us) server_negotiation.cc:407] Connection header received
0927 11:13:53.107953 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:53.107953 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:53.107963 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:53.108060 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:13:53.108620 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.109089 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.109794 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.109974 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.110087 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:53.111230 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:13:53.111231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:13:53.111231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:13:53.111237 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:13:53.112504 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:53.112836 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:53.112837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:53.112838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:53.112891 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:13:53.113101 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:13:53.113102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:13:53.113103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:13:53.113227 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:13:53.113331 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:53.113437 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 11:13:53.113507 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 11:13:53.115017 10630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:13:53.116362 10630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:13:53.121889 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:53.118164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44163 (local address 127.0.0.1:60304)
0927 11:13:53.118270 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:53.118305 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:53.118370 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:53.118669 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:53.118672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:53.118674 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:53.118729 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:53.118730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.120093 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.120094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:53.120907 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:53.120910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.121699 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.121700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:53.121746 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:53.121755 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 11:13:53.121799 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 11:13:53.121845 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":2,"thread_start_us":56,"threads_started":1}
I20260927 11:13:53.121930 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:53.118216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.0.98.3:44163)
0927 11:13:53.118504 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:53.118508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:53.118518 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:13:53.118552 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:53.118553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:53.118568 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:53.118652 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:53.118812 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.119977 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.121013 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.121629 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.121694 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:53.121736 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:53.121817 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 11:13:53.121864 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":72,"threads_started":1}
I20260927 11:13:53.122380 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:53.118164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42531 (local address 127.0.0.1:36458)
0927 11:13:53.118326 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:53.118336 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:53.118383 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:53.118720 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:53.118720 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:53.118722 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:53.118751 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:53.118752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.120182 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.120183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:53.121298 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:53.121300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.122172 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.122173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:53.122231 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:53.122234 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:53.122280 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 11:13:53.122313 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":11,"thread_start_us":51,"threads_started":1}
I20260927 11:13:53.122586 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:53.118379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.0.98.2:42531)
0927 11:13:53.118534 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:53.118538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:53.118547 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:13:53.118590 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:53.118591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:53.118602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:53.118825 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:53.118901 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.120095 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.121380 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.122095 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.122166 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:53.122217 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:53.122310 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 11:13:53.122350 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260927 11:13:53.122552 11023 tablet_service.cc:1467] Processing CreateTablet for tablet a590047111fa4402b78a6b5dd03d1542 (DEFAULT_TABLE table=test-table [id=165a518703464fb28faab96a415289dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:53.122862 11023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a590047111fa4402b78a6b5dd03d1542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:53.123297 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:53.118343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33097 (local address 127.0.0.1:45984)
0927 11:13:53.118714 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:53.118725 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:53.118770 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:53.118986 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:53.118987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:53.118989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:53.119036 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:53.119037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.120027 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.120028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:53.121297 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:53.121300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.123082 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.123083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:53.123141 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:53.123145 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:13:53.123194 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 11:13:53.123235 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":45,"thread_start_us":65,"threads_started":1}
I20260927 11:13:53.123229 10889 tablet_service.cc:1467] Processing CreateTablet for tablet a590047111fa4402b78a6b5dd03d1542 (DEFAULT_TABLE table=test-table [id=165a518703464fb28faab96a415289dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:53.123381 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:53.118555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.0.98.1:33097)
0927 11:13:53.118781 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:53.118784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:53.118794 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:13:53.118833 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:53.118835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:53.118847 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:53.118944 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:53.119118 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.119931 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.122253 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.122994 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.123072 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:53.123224 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:53.123254 (+ 30us) server_negotiation.cc:299] Negotiation successful
0927 11:13:53.123302 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":72,"threads_started":1}
I20260927 11:13:53.123515 10889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a590047111fa4402b78a6b5dd03d1542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:53.125213 11103 tablet_bootstrap.cc:492] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161: Bootstrap starting.
I20260927 11:13:53.125828 11103 tablet_bootstrap.cc:654] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:53.125913 11104 tablet_bootstrap.cc:492] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c: Bootstrap starting.
I20260927 11:13:53.126377 11103 log.cc:824] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:53.126612 10755 tablet_service.cc:1467] Processing CreateTablet for tablet a590047111fa4402b78a6b5dd03d1542 (DEFAULT_TABLE table=test-table [id=165a518703464fb28faab96a415289dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:13:53.126945 10755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a590047111fa4402b78a6b5dd03d1542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:13:53.127063 11103 tablet_bootstrap.cc:492] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161: No bootstrap required, opened a new log
I20260927 11:13:53.127125 11103 ts_tablet_manager.cc:1397] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 11:13:53.128636 11103 raft_consensus.cc:348] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c254c40149d94cdc96ec97319358f20c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42531 } } peers { permanent_uuid: "c9e9a874be3c423a80da9f2bef6d4161" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44163 } } peers { permanent_uuid: "c08dd3b140dc4c93bf686b726688c83a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33097 } }
I20260927 11:13:53.128757 11103 raft_consensus.cc:374] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:53.128778 11103 raft_consensus.cc:729] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e9a874be3c423a80da9f2bef6d4161, State: Initialized, Role: FOLLOWER
I20260927 11:13:53.128885 11103 consensus_queue.cc:260] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c254c40149d94cdc96ec97319358f20c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42531 } } peers { permanent_uuid: "c9e9a874be3c423a80da9f2bef6d4161" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44163 } } peers { permanent_uuid: "c08dd3b140dc4c93bf686b726688c83a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33097 } }
I20260927 11:13:53.129169 11089 heartbeater.cc:503] Master 127.0.98.62:45993 was elected leader, sending a full tablet report...
I20260927 11:13:53.129180 11103 ts_tablet_manager.cc:1428] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 11:13:53.129808 11107 tablet_bootstrap.cc:492] T a590047111fa4402b78a6b5dd03d1542 P c08dd3b140dc4c93bf686b726688c83a: Bootstrap starting.
I20260927 11:13:53.130174 11104 tablet_bootstrap.cc:654] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:53.130431 11107 tablet_bootstrap.cc:654] T a590047111fa4402b78a6b5dd03d1542 P c08dd3b140dc4c93bf686b726688c83a: Neither blocks nor log segments found. Creating new log.
I20260927 11:13:53.130708 11104 log.cc:824] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:53.130988 11107 log.cc:824] T a590047111fa4402b78a6b5dd03d1542 P c08dd3b140dc4c93bf686b726688c83a: Log is configured to *not* fsync() on all Append() calls
I20260927 11:13:53.131420 11104 tablet_bootstrap.cc:492] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c: No bootstrap required, opened a new log
I20260927 11:13:53.131488 11104 ts_tablet_manager.cc:1397] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 11:13:53.131611 11107 tablet_bootstrap.cc:492] T a590047111fa4402b78a6b5dd03d1542 P c08dd3b140dc4c93bf686b726688c83a: No bootstrap required, opened a new log
I20260927 11:13:53.131669 11107 ts_tablet_manager.cc:1397] T a590047111fa4402b78a6b5dd03d1542 P c08dd3b140dc4c93bf686b726688c83a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:13:53.133396 11104 raft_consensus.cc:348] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c254c40149d94cdc96ec97319358f20c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42531 } } peers { permanent_uuid: "c9e9a874be3c423a80da9f2bef6d4161" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44163 } } peers { permanent_uuid: "c08dd3b140dc4c93bf686b726688c83a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33097 } }
I20260927 11:13:53.133563 11104 raft_consensus.cc:374] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:53.133597 11104 raft_consensus.cc:729] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c254c40149d94cdc96ec97319358f20c, State: Initialized, Role: FOLLOWER
I20260927 11:13:53.133530 11107 raft_consensus.cc:348] T a590047111fa4402b78a6b5dd03d1542 P c08dd3b140dc4c93bf686b726688c83a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c254c40149d94cdc96ec97319358f20c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42531 } } peers { permanent_uuid: "c9e9a874be3c423a80da9f2bef6d4161" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44163 } } peers { permanent_uuid: "c08dd3b140dc4c93bf686b726688c83a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33097 } }
I20260927 11:13:53.133688 11107 raft_consensus.cc:374] T a590047111fa4402b78a6b5dd03d1542 P c08dd3b140dc4c93bf686b726688c83a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:13:53.133721 11107 raft_consensus.cc:729] T a590047111fa4402b78a6b5dd03d1542 P c08dd3b140dc4c93bf686b726688c83a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c08dd3b140dc4c93bf686b726688c83a, State: Initialized, Role: FOLLOWER
I20260927 11:13:53.133708 11104 consensus_queue.cc:260] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c254c40149d94cdc96ec97319358f20c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42531 } } peers { permanent_uuid: "c9e9a874be3c423a80da9f2bef6d4161" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44163 } } peers { permanent_uuid: "c08dd3b140dc4c93bf686b726688c83a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33097 } }
I20260927 11:13:53.133852 11107 consensus_queue.cc:260] T a590047111fa4402b78a6b5dd03d1542 P c08dd3b140dc4c93bf686b726688c83a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c254c40149d94cdc96ec97319358f20c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42531 } } peers { permanent_uuid: "c9e9a874be3c423a80da9f2bef6d4161" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44163 } } peers { permanent_uuid: "c08dd3b140dc4c93bf686b726688c83a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33097 } }
I20260927 11:13:53.133980 11104 ts_tablet_manager.cc:1428] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:13:53.134018 10955 heartbeater.cc:503] Master 127.0.98.62:45993 was elected leader, sending a full tablet report...
I20260927 11:13:53.134128 11107 ts_tablet_manager.cc:1428] T a590047111fa4402b78a6b5dd03d1542 P c08dd3b140dc4c93bf686b726688c83a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:13:53.134229 10821 heartbeater.cc:503] Master 127.0.98.62:45993 was elected leader, sending a full tablet report...
W20260927 11:13:53.201877 10956 tablet.cc:2369] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:13:53.308137 10822 tablet.cc:2369] T a590047111fa4402b78a6b5dd03d1542 P c08dd3b140dc4c93bf686b726688c83a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:13:53.340220 11090 tablet.cc:2369] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:13:53.343405 11110 raft_consensus.cc:482] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:13:53.343492 11110 raft_consensus.cc:504] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c254c40149d94cdc96ec97319358f20c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42531 } } peers { permanent_uuid: "c9e9a874be3c423a80da9f2bef6d4161" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44163 } } peers { permanent_uuid: "c08dd3b140dc4c93bf686b726688c83a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33097 } }
I20260927 11:13:53.343886 11110 leader_election.cc:288] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9e9a874be3c423a80da9f2bef6d4161 (127.0.98.3:44163), c08dd3b140dc4c93bf686b726688c83a (127.0.98.1:33097)
I20260927 11:13:53.346781 10957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:53.344051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44163 (local address 127.0.98.2:40149)
0927 11:13:53.344121 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:53.344133 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:53.344177 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:53.344376 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:53.344376 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:53.344378 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:53.344452 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:53.344454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.345094 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.345095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:53.345818 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:53.345820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.346520 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.346521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:53.346604 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:53.346607 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:53.346679 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 11:13:53.346719 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 11:13:53.346781 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:53.344164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40149 (local address 127.0.98.3:44163)
0927 11:13:53.344239 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:53.344241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:53.344251 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:13:53.344280 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:53.344281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:53.344292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:53.344362 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:53.344546 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.345016 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.345905 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.346463 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.346523 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:53.346532 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:53.346686 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 11:13:53.346719 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 11:13:53.346951 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:53.344266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36831 (local address 127.0.98.1:33097)
0927 11:13:53.344330 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:53.344332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:53.344427 (+ 95us) server_negotiation.cc:407] Connection header received
0927 11:13:53.344516 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:53.344517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:53.344534 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:53.344592 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:13:53.344748 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.345190 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.346119 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.346630 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.346703 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:53.346713 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:53.346863 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 11:13:53.346895 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 11:13:53.346958 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:53.344188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33097 (local address 127.0.98.2:36831)
0927 11:13:53.344369 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:53.344380 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:13:53.344435 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:13:53.344610 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:13:53.344611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:13:53.344613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:13:53.344680 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:53.344682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.345284 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.345285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:53.346001 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:13:53.346002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.346719 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.346720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:13:53.346786 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:53.346789 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:13:53.346846 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 11:13:53.346882 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":122,"threads_started":1}
I20260927 11:13:53.347165 11043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a590047111fa4402b78a6b5dd03d1542" candidate_uuid: "c254c40149d94cdc96ec97319358f20c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9a874be3c423a80da9f2bef6d4161" is_pre_election: true
I20260927 11:13:53.347307 11043 raft_consensus.cc:2454] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c254c40149d94cdc96ec97319358f20c in term 0.
I20260927 11:13:53.348273 10842 leader_election.cc:302] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c254c40149d94cdc96ec97319358f20c, c9e9a874be3c423a80da9f2bef6d4161; no voters:
I20260927 11:13:53.348397 11110 raft_consensus.cc:2781] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:13:53.348460 11110 raft_consensus.cc:482] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:13:53.348484 11110 raft_consensus.cc:3037] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:53.348605 10775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a590047111fa4402b78a6b5dd03d1542" candidate_uuid: "c254c40149d94cdc96ec97319358f20c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c08dd3b140dc4c93bf686b726688c83a" is_pre_election: true
I20260927 11:13:53.348743 10775 raft_consensus.cc:2454] T a590047111fa4402b78a6b5dd03d1542 P c08dd3b140dc4c93bf686b726688c83a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c254c40149d94cdc96ec97319358f20c in term 0.
I20260927 11:13:53.349274 11110 raft_consensus.cc:504] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c254c40149d94cdc96ec97319358f20c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42531 } } peers { permanent_uuid: "c9e9a874be3c423a80da9f2bef6d4161" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44163 } } peers { permanent_uuid: "c08dd3b140dc4c93bf686b726688c83a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33097 } }
I20260927 11:13:53.349392 11110 leader_election.cc:288] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [CANDIDATE]: Term 1 election: Requested vote from peers c9e9a874be3c423a80da9f2bef6d4161 (127.0.98.3:44163), c08dd3b140dc4c93bf686b726688c83a (127.0.98.1:33097)
I20260927 11:13:53.349556 11043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a590047111fa4402b78a6b5dd03d1542" candidate_uuid: "c254c40149d94cdc96ec97319358f20c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9e9a874be3c423a80da9f2bef6d4161"
I20260927 11:13:53.349557 10775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a590047111fa4402b78a6b5dd03d1542" candidate_uuid: "c254c40149d94cdc96ec97319358f20c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c08dd3b140dc4c93bf686b726688c83a"
I20260927 11:13:53.349622 11043 raft_consensus.cc:3037] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:53.349644 10775 raft_consensus.cc:3037] T a590047111fa4402b78a6b5dd03d1542 P c08dd3b140dc4c93bf686b726688c83a [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:13:53.350337 11043 raft_consensus.cc:2454] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c254c40149d94cdc96ec97319358f20c in term 1.
I20260927 11:13:53.350360 10775 raft_consensus.cc:2454] T a590047111fa4402b78a6b5dd03d1542 P c08dd3b140dc4c93bf686b726688c83a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c254c40149d94cdc96ec97319358f20c in term 1.
I20260927 11:13:53.350497 10842 leader_election.cc:302] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c254c40149d94cdc96ec97319358f20c, c9e9a874be3c423a80da9f2bef6d4161; no voters:
I20260927 11:13:53.350606 11110 raft_consensus.cc:2781] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:13:53.350765 11110 raft_consensus.cc:686] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [term 1 LEADER]: Becoming Leader. State: Replica: c254c40149d94cdc96ec97319358f20c, State: Running, Role: LEADER
I20260927 11:13:53.350864 11110 consensus_queue.cc:237] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c254c40149d94cdc96ec97319358f20c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42531 } } peers { permanent_uuid: "c9e9a874be3c423a80da9f2bef6d4161" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44163 } } peers { permanent_uuid: "c08dd3b140dc4c93bf686b726688c83a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33097 } }
I20260927 11:13:53.351624 10630 catalog_manager.cc:5495] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c reported cstate change: term changed from 0 to 1, leader changed from <none> to c254c40149d94cdc96ec97319358f20c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c254c40149d94cdc96ec97319358f20c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c254c40149d94cdc96ec97319358f20c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9e9a874be3c423a80da9f2bef6d4161" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c08dd3b140dc4c93bf686b726688c83a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33097 } health_report { overall_health: UNKNOWN } } }
I20260927 11:13:53.406241 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:53.404251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.0.98.2:42531)
0927 11:13:53.404314 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:13:53.404315 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:13:53.404322 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:13:53.404350 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:13:53.404351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:13:53.404367 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:13:53.404431 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:13:53.404530 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.404966 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.405641 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:13:53.405810 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:13:53.405856 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:13:53.405985 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:13:53.406089 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:13:53.406147 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 11:13:53.406160 (+ 13us) server_negotiation.cc:299] Negotiation successful
0927 11:13:53.406174 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 11:13:53.407886 10775 raft_consensus.cc:1261] T a590047111fa4402b78a6b5dd03d1542 P c08dd3b140dc4c93bf686b726688c83a [term 1 FOLLOWER]: Refusing update from remote peer c254c40149d94cdc96ec97319358f20c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:53.407915 11043 raft_consensus.cc:1261] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161 [term 1 FOLLOWER]: Refusing update from remote peer c254c40149d94cdc96ec97319358f20c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:13:53.408273 11113 consensus_queue.cc:1035] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9e9a874be3c423a80da9f2bef6d4161" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:53.408398 11110 consensus_queue.cc:1035] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c08dd3b140dc4c93bf686b726688c83a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:13:53.412482 10630 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:53.412566 10630 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:13:53.413264 10630 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cde68b748174eccbda61287f7eff209: Sending DeleteTablet for 3 replicas of tablet a590047111fa4402b78a6b5dd03d1542
I20260927 11:13:53.413623 10889 tablet_service.cc:1514] Processing DeleteTablet for tablet a590047111fa4402b78a6b5dd03d1542 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36458
I20260927 11:13:53.413641 11023 tablet_service.cc:1514] Processing DeleteTablet for tablet a590047111fa4402b78a6b5dd03d1542 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60304
I20260927 11:13:53.413887 11126 tablet_replica.cc:330] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c: stopping tablet replica
I20260927 11:13:53.413890 10755 tablet_service.cc:1514] Processing DeleteTablet for tablet a590047111fa4402b78a6b5dd03d1542 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:13:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45984
I20260927 11:13:53.414095 11126 raft_consensus.cc:2229] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:13:53.414170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 10692
I20260927 11:13:53.414352 11126 raft_consensus.cc:2258] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:53.414713 11126 ts_tablet_manager.cc:1905] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:53.416517 11126 ts_tablet_manager.cc:1918] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:53.416576 11126 log.cc:1196] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/a590047111fa4402b78a6b5dd03d1542
I20260927 11:13:53.416843 11126 ts_tablet_manager.cc:1939] T a590047111fa4402b78a6b5dd03d1542 P c254c40149d94cdc96ec97319358f20c: Deleting consensus metadata
I20260927 11:13:53.417092 11127 tablet_replica.cc:330] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161: stopping tablet replica
I20260927 11:13:53.417223 11127 raft_consensus.cc:2229] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:13:53.417235 10617 catalog_manager.cc:4841] TS c254c40149d94cdc96ec97319358f20c (127.0.98.2:42531): tablet a590047111fa4402b78a6b5dd03d1542 (table test-table [id=165a518703464fb28faab96a415289dc]) successfully deleted
I20260927 11:13:53.417325 11127 raft_consensus.cc:2258] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:13:53.417711 11127 ts_tablet_manager.cc:1905] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:13:53.419557 11127 ts_tablet_manager.cc:1918] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:13:53.419631 11127 log.cc:1196] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/a590047111fa4402b78a6b5dd03d1542
I20260927 11:13:53.419881 11127 ts_tablet_manager.cc:1939] T a590047111fa4402b78a6b5dd03d1542 P c9e9a874be3c423a80da9f2bef6d4161: Deleting consensus metadata
I20260927 11:13:53.420265 10614 catalog_manager.cc:4841] TS c9e9a874be3c423a80da9f2bef6d4161 (127.0.98.3:44163): tablet a590047111fa4402b78a6b5dd03d1542 (table test-table [id=165a518703464fb28faab96a415289dc]) successfully deleted
W20260927 11:13:53.421176 10614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33097 (error 108)
I20260927 11:13:53.421443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 10826
I20260927 11:13:53.421732 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:13:53.421617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33097 (local address 127.0.0.1:46000)
0927 11:13:53.421661 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 11:13:53.421696 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 11:13:53.422091 10614 catalog_manager.cc:4584] TS c08dd3b140dc4c93bf686b726688c83a (127.0.98.1:33097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a590047111fa4402b78a6b5dd03d1542: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33097: connect: Connection refused (error 111)
I20260927 11:13:53.426430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 10960
I20260927 11:13:53.431617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 10598
2026-09-27T11:13:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2052 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:13:53 dist-test-slave-9hq9 krb5kdc[11132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:13:53 dist-test-slave-9hq9 krb5kdc[11132](info): set up 2 sockets
Sep 27 11:13:53 dist-test-slave-9hq9 krb5kdc[11132](info): commencing operation
krb5kdc: starting...
W20260927 11:13:54.451093 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:13:54 dist-test-slave-9hq9 krb5kdc[11132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:13:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:13:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 11:13:54.602098 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGpH_nd/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/postgres -l logfile start
2026-09-27 11:13:56.915 UTC [11162] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 11:13:56.915 UTC [11162] LOG: listening on IPv4 address "127.0.98.20", port 58381
2026-09-27 11:13:56.918 UTC [11162] LOG: listening on Unix socket "/tmp/.s.PGSQL.58381"
2026-09-27 11:13:56.951 UTC [11165] LOG: database system was shut down at 2026-09-27 11:13:55 UTC
2026-09-27 11:13:56.955 UTC [11162] LOG: database system is ready to accept connections
I20260927 11:13:57.878753 392 mini_postgres.cc:96] Postgres bound to 58381
2026-09-27 11:13:57.884 UTC [11173] FATAL: database "slave" does not exist
127.0.98.20:58381 - accepting connections
I20260927 11:13:57.884865 392 mini_ranger.cc:161] Starting Ranger...
I20260927 11:13:57.898043 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 11:13:58.308648 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 11:13:58.308712 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-admin
2026-09-27 11:13:58,568 [I] DB FLAVOR :POSTGRES
2026-09-27 11:13:58,569 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 11:13:58,569 [I] Checking connection..
2026-09-27 11:13:58,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 11:13:58,788 [I] Checking connection passed.
2026-09-27 11:13:58,788 [I] --------- Verifying version history table ---------
2026-09-27 11:13:58,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 11:13:59,017 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 11:13:59,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 11:13:59,258 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 11:13:59,259 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 11:13:59,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 11:13:59.495 UTC [11237] WARNING: there is no transaction in progress
2026-09-27 11:13:59,500 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 11:13:59,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 11:13:59,717 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 11:13:59,717 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 11:13:59,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 11:13:59,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 11:14:00,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9hq9.c.gcp-upstream.internal','N') ;"
2026-09-27 11:14:00,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 11:14:00,600 [I] Table x_portal_user does not exist in database ranger
2026-09-27 11:14:00,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 11:14:00,819 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 11:14:00,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 11:14:01,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 11:14:01,264 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 11:14:01,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 11:14:02.113 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02.129 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02.143 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02.157 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02.172 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02.190 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02.249 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02.255 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02.267 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02.276 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02.286 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02.297 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02.306 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02.311 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02.317 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02.323 UTC [11345] WARNING: there is no transaction in progress
2026-09-27 11:14:02,894 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 11:14:02,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 11:14:03,135 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 11:14:03,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 11:14:03,373 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 11:14:03,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 11:14:03,597 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 11:14:03,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 11:14:03,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 11:14:04,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9hq9.c.gcp-upstream.internal';"
2026-09-27 11:14:04,259 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 11:14:04,260 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 11:14:04,481 [I] DB_PATCHES have already been applied
I20260927 11:14:04.487273 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGpH_nd/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 11:14:04.487332 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 11:14:04.490156 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:53271
Sep 27, 2026 11:14:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 11:14:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 11:14:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 11:14:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 11:14:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 11:14:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 11:14:05 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 11:14:05.584779 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.700s user 0.000s sys 0.008s
Sep 27, 2026 11:14:24 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 11:14:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 11:14:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 11:14:24 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 11:14:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 11:14:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 11:14:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 11:14:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 11:14:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 11:14:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 11:14:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 11:14:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 11:14:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 11:14:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 11:14:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 11:14:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 11:14:24 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 11:14:24 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 11:14:25.419456 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 11:14:25 dist-test-slave-9hq9 krb5kdc[11132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507665, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 11:14:25.508028 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 11:14:25.516798 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 11:14:26.294695 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 11:14:26.294780 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-kms
I20260927 11:14:26.294800 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:58381
2026-09-27 11:14:26,474 [I] DB FLAVOR :POSTGRES
2026-09-27 11:14:26,474 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 11:14:26,475 [I] Checking connection
2026-09-27 11:14:26,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 11:14:26,709 [I] connection success
2026-09-27 11:14:26,710 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 11:14:26,710 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 11:14:26,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 11:14:26,944 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 11:14:26,944 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 11:14:26,944 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 11:14:26,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 11:14:27,182 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 11:14:27.603211 392 mini_ranger_kms.cc:310] Created kms service
I20260927 11:14:27.806938 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 11:14:27.848652 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 11:14:27 dist-test-slave-9hq9 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507665, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:14:28.209733 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 11:14:28.209991 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 11:14:28.210032 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 11:14:28.212942 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11525
I20260927 11:14:28.213054 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:40321
Sep 27, 2026 11:14:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 11:14:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 11:14:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 11:14:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 11:14:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 11:14:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 11:14:30.388842 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.881s user 0.012s sys 0.150s
I20260927 11:14:30.388945 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 11:14:30.389029 392 mini_ranger_kms.cc:401] 127.0.98.20:40321/kms/v1/keys
Sep 27 11:14:30 dist-test-slave-9hq9 krb5kdc[11132](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790507670, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 11:14:30 dist-test-slave-9hq9 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507670, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 11:14:31 dist-test-slave-9hq9 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507670, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 11:14:33 dist-test-slave-9hq9 krb5kdc[11132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:14:33.312678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34591
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34591
--ranger_config_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40321/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:14:33.404420 11576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:14:33.404597 11576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:14:33.404618 11576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:14:33.405747 11576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:14:33.405786 11576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:14:33.405800 11576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:14:33.405812 11576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:14:33.405824 11576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:14:33.407370 11576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:40321/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34591
--ranger_config_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:14:33.407583 11576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:14:33.407807 11576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:14:33.413614 11576 server_base.cc:1034] running on GCE node
W20260927 11:14:33.413669 11581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:14:33.413830 11584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:14:33.413803 11582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:14:33.414220 11576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:14:33.414551 11576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:14:33.415727 11576 hybrid_clock.cc:648] HybridClock initialized: now 1790507673415712 us; error 27 us; skew 500 ppm
Sep 27 11:14:33 dist-test-slave-9hq9 krb5kdc[11132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:14:33.421276 11576 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:14:33.421757 11576 webserver.cc:460] Webserver started at http://127.0.98.62:36443/ using document root <none> and password file <none>
I20260927 11:14:33.421978 11576 fs_manager.cc:359] Metadata directory not provided
I20260927 11:14:33.422034 11576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:14:33.422196 11576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:14:33 dist-test-slave-9hq9 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:14:33.481618 11576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "021f40d35aac4a00b961262440c91557"
format_stamp: "Formatted at 2026-09-27 11:14:33 on dist-test-slave-9hq9"
server_key: "86dcae2bdf154bd211858dd4ea271330"
server_key_iv: "c81b50d2a2a0dd4d2d28b3f51a1e75c0"
server_key_version: "kuduclusterkey@0"
I20260927 11:14:33.482060 11576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "021f40d35aac4a00b961262440c91557"
format_stamp: "Formatted at 2026-09-27 11:14:33 on dist-test-slave-9hq9"
server_key: "86dcae2bdf154bd211858dd4ea271330"
server_key_iv: "c81b50d2a2a0dd4d2d28b3f51a1e75c0"
server_key_version: "kuduclusterkey@0"
I20260927 11:14:33.483476 11576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:14:33.497056 11591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:14:33.497211 11576 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:14:33.497295 11576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "021f40d35aac4a00b961262440c91557"
format_stamp: "Formatted at 2026-09-27 11:14:33 on dist-test-slave-9hq9"
server_key: "86dcae2bdf154bd211858dd4ea271330"
server_key_iv: "c81b50d2a2a0dd4d2d28b3f51a1e75c0"
server_key_version: "kuduclusterkey@0"
I20260927 11:14:33.497391 11576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:14:33.525225 11576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:14:33.527274 11576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:14:33.527429 11576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:14:33.532271 11576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34591
I20260927 11:14:33.532302 11643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34591 every 8 connection(s)
I20260927 11:14:33.532640 11576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:14:33.533274 11644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:14:33.535538 11644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557: Bootstrap starting.
I20260927 11:14:33.536116 11644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557: Neither blocks nor log segments found. Creating new log.
I20260927 11:14:33.536387 11644 log.cc:824] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557: Log is configured to *not* fsync() on all Append() calls
I20260927 11:14:33.537043 11644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557: No bootstrap required, opened a new log
I20260927 11:14:33.538960 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 11576
I20260927 11:14:33.539058 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:14:33.539247 392 external_mini_cluster.cc:1428] Setting key acf68401f53f61f83bafa7fec00d391a
I20260927 11:14:33.539639 11644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021f40d35aac4a00b961262440c91557" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34591 } }
I20260927 11:14:33.539902 11644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:14:33.539971 11644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 021f40d35aac4a00b961262440c91557, State: Initialized, Role: FOLLOWER
I20260927 11:14:33.540108 11644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021f40d35aac4a00b961262440c91557" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34591 } }
I20260927 11:14:33.540189 11644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:14:33.540217 11644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:14:33.540288 11644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:14:33.540859 11644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021f40d35aac4a00b961262440c91557" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34591 } }
I20260927 11:14:33.540973 11644 leader_election.cc:302] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 021f40d35aac4a00b961262440c91557; no voters:
I20260927 11:14:33.541136 11644 leader_election.cc:288] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:14:33.541186 11649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:14:33.541380 11649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [term 1 LEADER]: Becoming Leader. State: Replica: 021f40d35aac4a00b961262440c91557, State: Running, Role: LEADER
I20260927 11:14:33.541402 11644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:14:33.541520 11649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021f40d35aac4a00b961262440c91557" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34591 } }
I20260927 11:14:33.543421 11650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "021f40d35aac4a00b961262440c91557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021f40d35aac4a00b961262440c91557" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34591 } } }
I20260927 11:14:33.543495 11650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [sys.catalog]: This master's current role is: LEADER
I20260927 11:14:33.543617 11650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 021f40d35aac4a00b961262440c91557. Latest consensus state: current_term: 1 leader_uuid: "021f40d35aac4a00b961262440c91557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "021f40d35aac4a00b961262440c91557" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34591 } } }
I20260927 11:14:33.543664 11650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557 [sys.catalog]: This master's current role is: LEADER
I20260927 11:14:33.543689 11663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:14:33.544170 11663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:14:33.544274 11644 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 11:14:33.545760 11663 catalog_manager.cc:1347] Generated new cluster ID: 230a04dff8234ce08c03e90924c47dd2
I20260927 11:14:33.545804 11663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:14:33 dist-test-slave-9hq9 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:14:33.554013 11648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:33.539735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.0.98.62:34591)
0927 11:14:33.539990 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 11:14:33.539999 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:14:33.540029 (+ 30us) server_negotiation.cc:407] Connection header received
0927 11:14:33.540136 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:14:33.540143 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:14:33.540198 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:14:33.540382 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:14:33.541121 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:33.542722 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:33.543785 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:33.544389 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:33.544519 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:33.547392 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:14:33.547396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:14:33.547396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:14:33.547408 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:14:33.549357 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:14:33.549765 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:14:33.549768 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:14:33.549769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:14:33.549863 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:14:33.550080 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:14:33.550081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:14:33.550081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:14:33.550396 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:14:33.550522 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 11:14:33.550926 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 11:14:33.553366 (+ 2440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":81,"threads_started":1}
I20260927 11:14:33.568537 11663 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:14:33.569664 11663 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:14:33.576609 11663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557: Generated new TSK 0
I20260927 11:14:33.576822 11663 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:14:34.354936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34591
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:14:34.455806 11690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:14:34.455986 11690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:14:34.456007 11690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:14:34.457063 11690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:14:34.457101 11690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:14:34.457126 11690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:14:34.458779 11690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:40321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:14:34.458992 11690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:14:34.459281 11690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:14:34.465948 11695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:14:34.465971 11696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:14:34.465971 11698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:14:34.466542 11690 server_base.cc:1034] running on GCE node
I20260927 11:14:34.466744 11690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:14:34.467012 11690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:14:34.468187 11690 hybrid_clock.cc:648] HybridClock initialized: now 1790507674468166 us; error 34 us; skew 500 ppm
Sep 27 11:14:34 dist-test-slave-9hq9 krb5kdc[11132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:14:34.471297 11690 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:14:34.471906 11690 webserver.cc:460] Webserver started at http://127.0.98.1:45461/ using document root <none> and password file <none>
I20260927 11:14:34.472153 11690 fs_manager.cc:359] Metadata directory not provided
I20260927 11:14:34.472218 11690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:14:34.472343 11690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:14:34 dist-test-slave-9hq9 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:14:34.487627 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "721e73efda5749a983cc88711b0578ac"
format_stamp: "Formatted at 2026-09-27 11:14:34 on dist-test-slave-9hq9"
server_key: "fcc358a64c7bf9abbe5483f2836a5f77"
server_key_iv: "d2e82eb20c17808c90ddd2bd95bc6d58"
server_key_version: "kuduclusterkey@0"
I20260927 11:14:34.488050 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "721e73efda5749a983cc88711b0578ac"
format_stamp: "Formatted at 2026-09-27 11:14:34 on dist-test-slave-9hq9"
server_key: "fcc358a64c7bf9abbe5483f2836a5f77"
server_key_iv: "d2e82eb20c17808c90ddd2bd95bc6d58"
server_key_version: "kuduclusterkey@0"
I20260927 11:14:34.493566 11690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 11:14:34.509599 11707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:14:34.509778 11690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 11:14:34.509877 11690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "721e73efda5749a983cc88711b0578ac"
format_stamp: "Formatted at 2026-09-27 11:14:34 on dist-test-slave-9hq9"
server_key: "fcc358a64c7bf9abbe5483f2836a5f77"
server_key_iv: "d2e82eb20c17808c90ddd2bd95bc6d58"
server_key_version: "kuduclusterkey@0"
I20260927 11:14:34.509984 11690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:14:34.526526 11690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:14:34.528707 11690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:14:34.528821 11690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:14:34.529134 11690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:14:34.529439 11690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:14:34.529471 11690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:14:34.529497 11690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:14:34.529513 11690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:14:34.536366 11690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45331
I20260927 11:14:34.536741 11690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:14:34.539316 11820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45331 every 8 connection(s)
I20260927 11:14:34.544521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 11690
I20260927 11:14:34.544613 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:14:34.544792 392 external_mini_cluster.cc:1428] Setting key d6e9728c6651d381947ea9d8a940755d
Sep 27 11:14:34 dist-test-slave-9hq9 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:14:34.550974 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:34.540255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34591 (local address 127.0.98.1:50549)
0927 11:14:34.540516 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 11:14:34.540567 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 11:14:34.541142 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:14:34.541509 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:14:34.541513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:14:34.541733 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:14:34.542003 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:34.542009 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:34.543044 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:34.543045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:34.544328 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:34.544331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:34.545007 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:34.545008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:34.545141 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:34.545693 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:14:34.545702 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:14:34.547337 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:14:34.549309 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:14:34.549315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:14:34.549317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:14:34.549585 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:14:34.549796 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:14:34.549797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:14:34.549798 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:14:34.549966 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:14:34.550281 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:14:34.550287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:14:34.550457 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 11:14:34.550562 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 11:14:34.550718 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":94,"threads_started":1}
I20260927 11:14:34.551229 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:34.540221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50549 (local address 127.0.98.62:34591)
0927 11:14:34.540433 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 11:14:34.540435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:14:34.541267 (+ 832us) server_negotiation.cc:407] Connection header received
0927 11:14:34.541297 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:14:34.541298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:14:34.541311 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:14:34.541370 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:14:34.542122 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:34.542914 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:34.544426 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:34.544877 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:34.544944 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:34.547435 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:14:34.547436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:14:34.547437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:14:34.547443 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:14:34.549216 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:14:34.549670 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:14:34.549671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:14:34.549672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:14:34.549730 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:14:34.550041 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:14:34.550042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:14:34.550043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:14:34.550195 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:14:34.550255 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 11:14:34.551050 (+ 795us) server_negotiation.cc:299] Negotiation successful
0927 11:14:34.551136 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":95,"threads_started":1}
I20260927 11:14:34.553009 11821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34591
I20260927 11:14:34.553112 11821 heartbeater.cc:464] Registering TS with master...
I20260927 11:14:34.553278 11821 heartbeater.cc:511] Master 127.0.98.62:34591 requested a full tablet report, sending...
I20260927 11:14:34.553738 11608 ts_manager.cc:194] Registered new tserver with Master: 721e73efda5749a983cc88711b0578ac (127.0.98.1:45331)
I20260927 11:14:34.554617 11608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50549
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:14:34.582878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34591
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 11:14:34 dist-test-slave-9hq9 krb5kdc[11132](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790507674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 11:14:34.672490 11827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:14:34.672797 11827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:14:34.672869 11827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:14:34.674618 11827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:14:34.674721 11827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:14:34.674798 11827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:14:34.677031 11827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:40321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:14:34.677440 11827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:14:34.677734 11827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:14:34.684442 11835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:14:34.684703 11837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:14:34.684934 11834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:14:34.686053 11827 server_base.cc:1034] running on GCE node
I20260927 11:14:34.686260 11827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:14:34.686498 11827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:14:34.687704 11827 hybrid_clock.cc:648] HybridClock initialized: now 1790507674687670 us; error 46 us; skew 500 ppm
Sep 27 11:14:34 dist-test-slave-9hq9 krb5kdc[11132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:14:34.690944 11827 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:14:34.691392 11827 webserver.cc:460] Webserver started at http://127.0.98.2:36535/ using document root <none> and password file <none>
I20260927 11:14:34.691627 11827 fs_manager.cc:359] Metadata directory not provided
I20260927 11:14:34.691686 11827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:14:34.691802 11827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:14:34 dist-test-slave-9hq9 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:14:34.712924 11827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "8a32183ac1e44e0ab839ee84b5a85976"
format_stamp: "Formatted at 2026-09-27 11:14:34 on dist-test-slave-9hq9"
server_key: "3b2771db65bd37c505c65677a3f3370e"
server_key_iv: "45f6d60e9da46eb6e9449e65386e7312"
server_key_version: "kuduclusterkey@0"
I20260927 11:14:34.713431 11827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8a32183ac1e44e0ab839ee84b5a85976"
format_stamp: "Formatted at 2026-09-27 11:14:34 on dist-test-slave-9hq9"
server_key: "3b2771db65bd37c505c65677a3f3370e"
server_key_iv: "45f6d60e9da46eb6e9449e65386e7312"
server_key_version: "kuduclusterkey@0"
I20260927 11:14:34.718163 11827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
I20260927 11:14:34.734584 11844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:14:34.734802 11827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 11:14:34.734882 11827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "8a32183ac1e44e0ab839ee84b5a85976"
format_stamp: "Formatted at 2026-09-27 11:14:34 on dist-test-slave-9hq9"
server_key: "3b2771db65bd37c505c65677a3f3370e"
server_key_iv: "45f6d60e9da46eb6e9449e65386e7312"
server_key_version: "kuduclusterkey@0"
I20260927 11:14:34.734951 11827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:14:34.760198 11827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:14:34.762607 11827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:14:34.762799 11827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:14:34.763039 11827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:14:34.763424 11827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:14:34.763506 11827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:14:34.763578 11827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:14:34.763648 11827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:14:34.769835 11827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42155
I20260927 11:14:34.772086 11958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42155 every 8 connection(s)
I20260927 11:14:34.772408 11827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:14:34.778786 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 11827
I20260927 11:14:34.778868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:14:34.779042 392 external_mini_cluster.cc:1428] Setting key 110d5bf14f971def2fec7c5d89d91d24
Sep 27 11:14:34 dist-test-slave-9hq9 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:14:34.783496 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:34.773468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54801 (local address 127.0.98.62:34591)
0927 11:14:34.773517 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 11:14:34.773519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:14:34.773736 (+ 217us) server_negotiation.cc:407] Connection header received
0927 11:14:34.774413 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:14:34.774415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:14:34.774428 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:14:34.774459 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:14:34.775227 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:34.776010 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:34.777427 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:34.777755 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:34.777810 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:34.780282 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:14:34.780283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:14:34.780284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:14:34.780291 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:14:34.781975 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:14:34.782450 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:14:34.782451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:14:34.782451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:14:34.782506 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:14:34.782775 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:14:34.782776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:14:34.782777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:14:34.782893 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:14:34.782941 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:14:34.783321 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 11:14:34.783405 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:14:34.783962 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:34.773468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34591 (local address 127.0.98.2:54801)
0927 11:14:34.773613 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 11:14:34.773659 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 11:14:34.774286 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:14:34.774648 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:14:34.774652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:14:34.774871 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:14:34.775115 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:34.775121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:34.776121 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:34.776122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:34.777329 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:34.777331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:34.777891 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:34.777891 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:34.778028 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:34.778563 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:14:34.778573 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:14:34.780176 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:14:34.782095 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:14:34.782100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:14:34.782102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:14:34.782362 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:14:34.782572 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:14:34.782573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:14:34.782573 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:14:34.782705 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:14:34.782971 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:14:34.782976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:14:34.783148 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 11:14:34.783588 (+ 440us) client_negotiation.cc:240] Negotiation successful
0927 11:14:34.783755 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260927 11:14:34.784446 11959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34591
I20260927 11:14:34.784538 11959 heartbeater.cc:464] Registering TS with master...
I20260927 11:14:34.784729 11959 heartbeater.cc:511] Master 127.0.98.62:34591 requested a full tablet report, sending...
I20260927 11:14:34.785058 11608 ts_manager.cc:194] Registered new tserver with Master: 8a32183ac1e44e0ab839ee84b5a85976 (127.0.98.2:42155)
I20260927 11:14:34.785514 11608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54801
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:14:34.816515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34591
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:14:34.934151 11964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:14:34.934475 11964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:14:34.934556 11964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:14:34.936213 11964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:14:34.936329 11964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:14:34.936410 11964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:14:34.938478 11964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:40321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:14:34.938779 11964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:14:34.939009 11964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:14:34.945173 11972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:14:34.945634 11969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:14:34.946353 11970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:14:34.946609 11964 server_base.cc:1034] running on GCE node
I20260927 11:14:34.946801 11964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:14:34.947032 11964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:14:34.948187 11964 hybrid_clock.cc:648] HybridClock initialized: now 1790507674948166 us; error 33 us; skew 500 ppm
Sep 27 11:14:34 dist-test-slave-9hq9 krb5kdc[11132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:14:34.956681 11964 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:14:34.957720 11964 webserver.cc:460] Webserver started at http://127.0.98.3:34891/ using document root <none> and password file <none>
I20260927 11:14:34.958021 11964 fs_manager.cc:359] Metadata directory not provided
I20260927 11:14:34.958117 11964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:14:34.958276 11964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:14:34 dist-test-slave-9hq9 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:14:34.979254 11964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "9b881d26e126437ab1332fa6ee4a088b"
format_stamp: "Formatted at 2026-09-27 11:14:34 on dist-test-slave-9hq9"
server_key: "13c67ec8718e60048c44871df7bf0e28"
server_key_iv: "3cad1d90e47562aeaa5d101453e98a2c"
server_key_version: "kuduclusterkey@0"
I20260927 11:14:34.980603 11964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9b881d26e126437ab1332fa6ee4a088b"
format_stamp: "Formatted at 2026-09-27 11:14:34 on dist-test-slave-9hq9"
server_key: "13c67ec8718e60048c44871df7bf0e28"
server_key_iv: "3cad1d90e47562aeaa5d101453e98a2c"
server_key_version: "kuduclusterkey@0"
I20260927 11:14:34.990438 11964 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.001s sys 0.001s
I20260927 11:14:35.006341 11979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:14:35.006549 11964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:14:35.006624 11964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "9b881d26e126437ab1332fa6ee4a088b"
format_stamp: "Formatted at 2026-09-27 11:14:34 on dist-test-slave-9hq9"
server_key: "13c67ec8718e60048c44871df7bf0e28"
server_key_iv: "3cad1d90e47562aeaa5d101453e98a2c"
server_key_version: "kuduclusterkey@0"
I20260927 11:14:35.006697 11964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:14:35.026412 11964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:14:35.028690 11964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:14:35.028855 11964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:14:35.029083 11964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:14:35.029469 11964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:14:35.029544 11964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:14:35.029608 11964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:14:35.029662 11964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:14:35.036396 11964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32891
I20260927 11:14:35.036831 11964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:14:35.039017 12092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32891 every 8 connection(s)
I20260927 11:14:35.044816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 11964
I20260927 11:14:35.044898 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:14:35.045080 392 external_mini_cluster.cc:1428] Setting key 39ec54e25ba44a2ea66ead37dd952402
Sep 27 11:14:35 dist-test-slave-9hq9 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:14:35.050891 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.040467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39003 (local address 127.0.98.62:34591)
0927 11:14:35.040510 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:14:35.040512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:14:35.041021 (+ 509us) server_negotiation.cc:407] Connection header received
0927 11:14:35.041793 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:14:35.041795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:14:35.041820 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:14:35.041854 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:14:35.042709 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.043504 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.045011 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.045304 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.045346 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:35.047750 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:14:35.047751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:14:35.047752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:14:35.047758 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:14:35.049413 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:14:35.049876 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:14:35.049877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:14:35.049877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:14:35.049930 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:14:35.050216 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:14:35.050217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:14:35.050217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:14:35.050334 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:14:35.050375 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 11:14:35.050711 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 11:14:35.050790 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:14:35.051525 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.040467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34591 (local address 127.0.98.3:39003)
0927 11:14:35.040857 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 11:14:35.040900 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:14:35.041662 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:14:35.042099 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:14:35.042104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:14:35.042336 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:14:35.042596 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:35.042602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.043625 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.043626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:35.044895 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:35.044898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.045428 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.045429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:35.045561 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:35.046140 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:14:35.046149 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:14:35.047646 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:14:35.049506 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:14:35.049511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:14:35.049513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:14:35.049776 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:14:35.049996 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:14:35.049997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:14:35.049997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:14:35.050142 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:14:35.050418 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:14:35.050423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:14:35.050573 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 11:14:35.051187 (+ 614us) client_negotiation.cc:240] Negotiation successful
0927 11:14:35.051329 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":88,"threads_started":1}
I20260927 11:14:35.052121 12093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34591
I20260927 11:14:35.052203 12093 heartbeater.cc:464] Registering TS with master...
I20260927 11:14:35.052361 12093 heartbeater.cc:511] Master 127.0.98.62:34591 requested a full tablet report, sending...
I20260927 11:14:35.052734 11608 ts_manager.cc:194] Registered new tserver with Master: 9b881d26e126437ab1332fa6ee4a088b (127.0.98.3:32891)
I20260927 11:14:35.053223 11608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39003
I20260927 11:14:35.056625 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:14:35.069329 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.059523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39734 (local address 127.0.98.62:34591)
0927 11:14:35.059558 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 11:14:35.059560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:14:35.059816 (+ 256us) server_negotiation.cc:407] Connection header received
0927 11:14:35.059914 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:14:35.059915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:14:35.059925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:14:35.059953 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:14:35.061251 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.062019 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.064632 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.064937 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.064979 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:35.066453 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:14:35.066454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:14:35.066454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:14:35.066461 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:14:35.068155 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:14:35.068538 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:14:35.068539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:14:35.068540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:14:35.068590 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:14:35.068777 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:14:35.068778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:14:35.068778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:14:35.068910 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:14:35.068986 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 11:14:35.069180 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 11:14:35.069248 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:14:35.071417 11608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:14:35.073557 11608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:14:35.101794 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.085758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32891 (local address 127.0.0.1:42460)
0927 11:14:35.086970 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0927 11:14:35.087010 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:14:35.087068 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:14:35.090259 (+ 3191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:14:35.090263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:14:35.090265 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:14:35.090340 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:35.090342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.097142 (+ 6800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.097143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:35.100188 (+ 3045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:35.100191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.101448 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.101449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:35.101546 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:35.101563 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 11:14:35.101627 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 11:14:35.101687 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"thread_start_us":71,"threads_started":1}
I20260927 11:14:35.103446 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.086190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42460 (local address 127.0.98.3:32891)
0927 11:14:35.089470 (+ 3280us) server_negotiation.cc:206] Beginning negotiation
0927 11:14:35.089475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:14:35.089484 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:14:35.089522 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:14:35.089524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:14:35.089541 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:14:35.089630 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:14:35.095680 (+ 6050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.096794 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.100310 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.101148 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.101311 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:35.101419 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 11:14:35.103315 (+ 1896us) server_negotiation.cc:299] Negotiation successful
0927 11:14:35.103360 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3242,"thread_start_us":67,"threads_started":1}
I20260927 11:14:35.104303 12027 tablet_service.cc:1467] Processing CreateTablet for tablet 5235213d2edc4dc8b5bb84a830e01311 (DEFAULT_TABLE table=test-table [id=8e5bad07a9ed40d99d74870afa6a5e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:14:35.104646 12027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5235213d2edc4dc8b5bb84a830e01311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:14:35.111374 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.085969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42155 (local address 127.0.0.1:50652)
0927 11:14:35.089021 (+ 3052us) negotiation.cc:107] Waiting for socket to connect
0927 11:14:35.089031 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:14:35.089085 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:14:35.104054 (+ 14969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:14:35.104055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:14:35.104057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:14:35.104094 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:35.104096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.106534 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.106536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:35.107721 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:35.107724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.111155 (+ 3431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.111156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:35.111224 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:35.111228 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:14:35.111274 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 11:14:35.111309 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3033,"thread_start_us":59,"threads_started":1}
I20260927 11:14:35.111680 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.086425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45331 (local address 127.0.0.1:50278)
0927 11:14:35.090470 (+ 4045us) negotiation.cc:107] Waiting for socket to connect
0927 11:14:35.090481 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:14:35.090525 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:14:35.104177 (+ 13652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:14:35.104178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:14:35.104179 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:14:35.104231 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:35.104232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.107834 (+ 3602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.107835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:35.108899 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:35.108901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.111460 (+ 2559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.111461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:35.111536 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:35.111541 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:14:35.111577 (+ 36us) client_negotiation.cc:240] Negotiation successful
0927 11:14:35.111621 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4030,"thread_start_us":59,"threads_started":1}
I20260927 11:14:35.111912 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.086586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50278 (local address 127.0.98.1:45331)
0927 11:14:35.096937 (+ 10351us) server_negotiation.cc:206] Beginning negotiation
0927 11:14:35.096941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:14:35.096950 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:14:35.096981 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:14:35.096983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:14:35.096998 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:14:35.097063 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:14:35.104311 (+ 7248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.105282 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.108996 (+ 3714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.109708 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.109777 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:35.109826 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:14:35.111790 (+ 1964us) server_negotiation.cc:299] Negotiation successful
0927 11:14:35.111841 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10317,"thread_start_us":53,"threads_started":1}
I20260927 11:14:35.112146 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.086104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.0.98.2:42155)
0927 11:14:35.098185 (+ 12081us) server_negotiation.cc:206] Beginning negotiation
0927 11:14:35.098190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:14:35.098200 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:14:35.098239 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:14:35.098241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:14:35.098255 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:14:35.098343 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:14:35.105384 (+ 7041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.106440 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.109874 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.111001 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.111065 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:35.111123 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 11:14:35.112034 (+ 911us) server_negotiation.cc:299] Negotiation successful
0927 11:14:35.112073 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12042,"thread_start_us":90,"threads_started":1}
I20260927 11:14:35.112602 11744 tablet_service.cc:1467] Processing CreateTablet for tablet 5235213d2edc4dc8b5bb84a830e01311 (DEFAULT_TABLE table=test-table [id=8e5bad07a9ed40d99d74870afa6a5e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:14:35.112700 11893 tablet_service.cc:1467] Processing CreateTablet for tablet 5235213d2edc4dc8b5bb84a830e01311 (DEFAULT_TABLE table=test-table [id=8e5bad07a9ed40d99d74870afa6a5e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:14:35.112939 11744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5235213d2edc4dc8b5bb84a830e01311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:14:35.113001 11893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5235213d2edc4dc8b5bb84a830e01311. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:14:35.123778 12114 tablet_bootstrap.cc:492] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976: Bootstrap starting.
I20260927 11:14:35.124861 12114 tablet_bootstrap.cc:654] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976: Neither blocks nor log segments found. Creating new log.
I20260927 11:14:35.125260 12114 log.cc:824] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976: Log is configured to *not* fsync() on all Append() calls
I20260927 11:14:35.126016 12114 tablet_bootstrap.cc:492] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976: No bootstrap required, opened a new log
I20260927 11:14:35.126117 12114 ts_tablet_manager.cc:1397] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:14:35.127797 12114 raft_consensus.cc:348] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b881d26e126437ab1332fa6ee4a088b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "8a32183ac1e44e0ab839ee84b5a85976" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "721e73efda5749a983cc88711b0578ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45331 } }
I20260927 11:14:35.127959 12114 raft_consensus.cc:374] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:14:35.127995 12114 raft_consensus.cc:729] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a32183ac1e44e0ab839ee84b5a85976, State: Initialized, Role: FOLLOWER
I20260927 11:14:35.128121 12114 consensus_queue.cc:260] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b881d26e126437ab1332fa6ee4a088b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "8a32183ac1e44e0ab839ee84b5a85976" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "721e73efda5749a983cc88711b0578ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45331 } }
I20260927 11:14:35.128369 12114 ts_tablet_manager.cc:1428] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:14:35.128554 11959 heartbeater.cc:503] Master 127.0.98.62:34591 was elected leader, sending a full tablet report...
I20260927 11:14:35.135064 12112 tablet_bootstrap.cc:492] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b: Bootstrap starting.
I20260927 11:14:35.136107 12112 tablet_bootstrap.cc:654] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b: Neither blocks nor log segments found. Creating new log.
I20260927 11:14:35.136417 12112 log.cc:824] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b: Log is configured to *not* fsync() on all Append() calls
I20260927 11:14:35.136754 12113 tablet_bootstrap.cc:492] T 5235213d2edc4dc8b5bb84a830e01311 P 721e73efda5749a983cc88711b0578ac: Bootstrap starting.
I20260927 11:14:35.137434 12113 tablet_bootstrap.cc:654] T 5235213d2edc4dc8b5bb84a830e01311 P 721e73efda5749a983cc88711b0578ac: Neither blocks nor log segments found. Creating new log.
I20260927 11:14:35.137712 12113 log.cc:824] T 5235213d2edc4dc8b5bb84a830e01311 P 721e73efda5749a983cc88711b0578ac: Log is configured to *not* fsync() on all Append() calls
I20260927 11:14:35.138537 12116 raft_consensus.cc:482] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:14:35.138747 12116 raft_consensus.cc:504] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b881d26e126437ab1332fa6ee4a088b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "8a32183ac1e44e0ab839ee84b5a85976" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "721e73efda5749a983cc88711b0578ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45331 } }
I20260927 11:14:35.140224 12116 leader_election.cc:288] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b881d26e126437ab1332fa6ee4a088b (127.0.98.3:32891), 721e73efda5749a983cc88711b0578ac (127.0.98.1:45331)
I20260927 11:14:35.141302 12112 tablet_bootstrap.cc:492] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b: No bootstrap required, opened a new log
I20260927 11:14:35.141373 12112 ts_tablet_manager.cc:1397] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 11:14:35.143507 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.139247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32891 (local address 127.0.98.2:40537)
0927 11:14:35.139288 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 11:14:35.139299 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:14:35.139340 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:14:35.139494 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:14:35.139495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:14:35.139496 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:14:35.139560 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:35.139562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.140924 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.140926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:35.142275 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:35.142278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.143232 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.143233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:35.143318 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:35.143322 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:14:35.143376 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 11:14:35.143423 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 11:14:35.143702 12112 raft_consensus.cc:348] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b881d26e126437ab1332fa6ee4a088b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "8a32183ac1e44e0ab839ee84b5a85976" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "721e73efda5749a983cc88711b0578ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45331 } }
I20260927 11:14:35.143872 12112 raft_consensus.cc:374] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:14:35.143905 12112 raft_consensus.cc:729] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b881d26e126437ab1332fa6ee4a088b, State: Initialized, Role: FOLLOWER
I20260927 11:14:35.145027 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.139344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40537 (local address 127.0.98.3:32891)
0927 11:14:35.139381 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 11:14:35.139383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:14:35.139390 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:14:35.139426 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:14:35.139427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:14:35.139436 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:14:35.139492 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:14:35.139653 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.140806 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.142449 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.143111 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.143185 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:35.143198 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0927 11:14:35.144902 (+ 1704us) server_negotiation.cc:299] Negotiation successful
0927 11:14:35.144952 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:14:35.145262 12044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5235213d2edc4dc8b5bb84a830e01311" candidate_uuid: "8a32183ac1e44e0ab839ee84b5a85976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b881d26e126437ab1332fa6ee4a088b" is_pre_election: true
I20260927 11:14:35.147727 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.139690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45331 (local address 127.0.98.2:42349)
0927 11:14:35.143575 (+ 3885us) negotiation.cc:107] Waiting for socket to connect
0927 11:14:35.143586 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:14:35.143645 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:14:35.143952 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:14:35.143953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:14:35.143954 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:14:35.143989 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:35.143990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.145479 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.145480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:35.146652 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:14:35.146654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.147550 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.147551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:14:35.147597 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:35.147600 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:14:35.147639 (+ 39us) client_negotiation.cc:240] Negotiation successful
0927 11:14:35.147669 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3874,"thread_start_us":502,"threads_started":1}
I20260927 11:14:35.148010 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.142160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42349 (local address 127.0.98.1:45331)
0927 11:14:35.142392 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 11:14:35.142394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:14:35.143772 (+ 1378us) server_negotiation.cc:407] Connection header received
0927 11:14:35.143802 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:14:35.143803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:14:35.143815 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:14:35.143866 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:14:35.144069 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.144802 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.146829 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.147471 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.147541 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:35.147550 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:14:35.147916 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 11:14:35.147957 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207}
I20260927 11:14:35.148458 11773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5235213d2edc4dc8b5bb84a830e01311" candidate_uuid: "8a32183ac1e44e0ab839ee84b5a85976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721e73efda5749a983cc88711b0578ac" is_pre_election: true
W20260927 11:14:35.150295 11846 leader_election.cc:341] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 721e73efda5749a983cc88711b0578ac (127.0.98.1:45331): Illegal state: must be running to vote when last-logged opid is not known
I20260927 11:14:35.150667 12112 consensus_queue.cc:260] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b881d26e126437ab1332fa6ee4a088b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "8a32183ac1e44e0ab839ee84b5a85976" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "721e73efda5749a983cc88711b0578ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45331 } }
I20260927 11:14:35.150892 12044 raft_consensus.cc:2454] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a32183ac1e44e0ab839ee84b5a85976 in term 0.
I20260927 11:14:35.151029 12112 ts_tablet_manager.cc:1428] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20260927 11:14:35.151081 12093 heartbeater.cc:503] Master 127.0.98.62:34591 was elected leader, sending a full tablet report...
I20260927 11:14:35.151350 11845 leader_election.cc:302] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a32183ac1e44e0ab839ee84b5a85976, 9b881d26e126437ab1332fa6ee4a088b; no voters: 721e73efda5749a983cc88711b0578ac
I20260927 11:14:35.151554 12116 raft_consensus.cc:2781] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:14:35.151629 12116 raft_consensus.cc:482] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:14:35.151659 12116 raft_consensus.cc:3037] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:14:35.152416 12116 raft_consensus.cc:504] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b881d26e126437ab1332fa6ee4a088b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "8a32183ac1e44e0ab839ee84b5a85976" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "721e73efda5749a983cc88711b0578ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45331 } }
I20260927 11:14:35.152540 12116 leader_election.cc:288] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [CANDIDATE]: Term 1 election: Requested vote from peers 9b881d26e126437ab1332fa6ee4a088b (127.0.98.3:32891), 721e73efda5749a983cc88711b0578ac (127.0.98.1:45331)
I20260927 11:14:35.152768 11773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5235213d2edc4dc8b5bb84a830e01311" candidate_uuid: "8a32183ac1e44e0ab839ee84b5a85976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "721e73efda5749a983cc88711b0578ac"
I20260927 11:14:35.152801 12044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5235213d2edc4dc8b5bb84a830e01311" candidate_uuid: "8a32183ac1e44e0ab839ee84b5a85976" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b881d26e126437ab1332fa6ee4a088b"
I20260927 11:14:35.152868 12044 raft_consensus.cc:3037] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b [term 0 FOLLOWER]: Advancing to term 1
W20260927 11:14:35.152987 11846 leader_election.cc:341] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 721e73efda5749a983cc88711b0578ac (127.0.98.1:45331): Illegal state: must be running to vote when last-logged opid is not known
I20260927 11:14:35.153617 12044 raft_consensus.cc:2454] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a32183ac1e44e0ab839ee84b5a85976 in term 1.
I20260927 11:14:35.153776 11845 leader_election.cc:302] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8a32183ac1e44e0ab839ee84b5a85976, 9b881d26e126437ab1332fa6ee4a088b; no voters: 721e73efda5749a983cc88711b0578ac
I20260927 11:14:35.153851 12116 raft_consensus.cc:2781] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:14:35.153970 12116 raft_consensus.cc:686] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [term 1 LEADER]: Becoming Leader. State: Replica: 8a32183ac1e44e0ab839ee84b5a85976, State: Running, Role: LEADER
I20260927 11:14:35.154067 12116 consensus_queue.cc:237] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b881d26e126437ab1332fa6ee4a088b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "8a32183ac1e44e0ab839ee84b5a85976" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "721e73efda5749a983cc88711b0578ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45331 } }
I20260927 11:14:35.154716 11608 catalog_manager.cc:5495] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a32183ac1e44e0ab839ee84b5a85976 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8a32183ac1e44e0ab839ee84b5a85976" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b881d26e126437ab1332fa6ee4a088b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a32183ac1e44e0ab839ee84b5a85976" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "721e73efda5749a983cc88711b0578ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45331 } health_report { overall_health: UNKNOWN } } }
I20260927 11:14:35.163280 12113 tablet_bootstrap.cc:492] T 5235213d2edc4dc8b5bb84a830e01311 P 721e73efda5749a983cc88711b0578ac: No bootstrap required, opened a new log
I20260927 11:14:35.163353 12113 ts_tablet_manager.cc:1397] T 5235213d2edc4dc8b5bb84a830e01311 P 721e73efda5749a983cc88711b0578ac: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.000s
I20260927 11:14:35.164598 12113 raft_consensus.cc:348] T 5235213d2edc4dc8b5bb84a830e01311 P 721e73efda5749a983cc88711b0578ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b881d26e126437ab1332fa6ee4a088b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "8a32183ac1e44e0ab839ee84b5a85976" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "721e73efda5749a983cc88711b0578ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45331 } }
I20260927 11:14:35.164740 12113 raft_consensus.cc:374] T 5235213d2edc4dc8b5bb84a830e01311 P 721e73efda5749a983cc88711b0578ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:14:35.164776 12113 raft_consensus.cc:729] T 5235213d2edc4dc8b5bb84a830e01311 P 721e73efda5749a983cc88711b0578ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721e73efda5749a983cc88711b0578ac, State: Initialized, Role: FOLLOWER
I20260927 11:14:35.164876 12113 consensus_queue.cc:260] T 5235213d2edc4dc8b5bb84a830e01311 P 721e73efda5749a983cc88711b0578ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b881d26e126437ab1332fa6ee4a088b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 } } peers { permanent_uuid: "8a32183ac1e44e0ab839ee84b5a85976" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42155 } } peers { permanent_uuid: "721e73efda5749a983cc88711b0578ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45331 } }
I20260927 11:14:35.165122 12113 ts_tablet_manager.cc:1428] T 5235213d2edc4dc8b5bb84a830e01311 P 721e73efda5749a983cc88711b0578ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:14:35.165251 11821 heartbeater.cc:503] Master 127.0.98.62:34591 was elected leader, sending a full tablet report...
I20260927 11:14:35.171052 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.167707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50654 (local address 127.0.98.2:42155)
0927 11:14:35.167746 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 11:14:35.167748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:14:35.167758 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:14:35.167847 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:14:35.167847 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:14:35.167866 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:14:35.167930 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:14:35.168123 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.168880 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.169986 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:14:35.170305 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:14:35.170363 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:14:35.170539 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:14:35.170665 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:14:35.170731 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 11:14:35.170945 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 11:14:35.170964 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 27 11:14:35 dist-test-slave-9hq9 krb5kdc[11132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:14:35.178586 11773 raft_consensus.cc:3037] T 5235213d2edc4dc8b5bb84a830e01311 P 721e73efda5749a983cc88711b0578ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:14:35.178674 12044 raft_consensus.cc:1261] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b [term 1 FOLLOWER]: Refusing update from remote peer 8a32183ac1e44e0ab839ee84b5a85976: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:14:35.178984 12121 consensus_queue.cc:1035] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b881d26e126437ab1332fa6ee4a088b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:14:35.179626 11773 raft_consensus.cc:1261] T 5235213d2edc4dc8b5bb84a830e01311 P 721e73efda5749a983cc88711b0578ac [term 1 FOLLOWER]: Refusing update from remote peer 8a32183ac1e44e0ab839ee84b5a85976: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:14:35.180080 12121 consensus_queue.cc:1035] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [LEADER]: Connected to new peer: Peer: permanent_uuid: "721e73efda5749a983cc88711b0578ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:14:35.188148 12125 mvcc.cc:205] Tried to move back new op lower bound from 7333919437520244736 to 7333919437431599104. Current Snapshot: MvccSnapshot[applied={T|T < 7333919437520244736}]
I20260927 11:14:35.188726 12128 mvcc.cc:205] Tried to move back new op lower bound from 7333919437520244736 to 7333919437431599104. Current Snapshot: MvccSnapshot[applied={T|T < 7333919437520244736}]
I20260927 11:14:35.190693 12126 mvcc.cc:205] Tried to move back new op lower bound from 7333919437520244736 to 7333919437431599104. Current Snapshot: MvccSnapshot[applied={T|T < 7333919437520244736}]
I20260927 11:14:35.193243 11608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:14:35.193346 11608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:14:35.194031 11608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 021f40d35aac4a00b961262440c91557: Sending DeleteTablet for 3 replicas of tablet 5235213d2edc4dc8b5bb84a830e01311
I20260927 11:14:35.194578 12027 tablet_service.cc:1514] Processing DeleteTablet for tablet 5235213d2edc4dc8b5bb84a830e01311 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:14:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42460
I20260927 11:14:35.194798 11893 tablet_service.cc:1514] Processing DeleteTablet for tablet 5235213d2edc4dc8b5bb84a830e01311 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:14:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50652
I20260927 11:14:35.194978 11744 tablet_service.cc:1514] Processing DeleteTablet for tablet 5235213d2edc4dc8b5bb84a830e01311 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:14:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50278
I20260927 11:14:35.195469 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 11690
I20260927 11:14:35.200879 12135 tablet_replica.cc:330] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976: stopping tablet replica
I20260927 11:14:35.206300 12135 raft_consensus.cc:2229] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:14:35.206481 12135 raft_consensus.cc:2258] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:14:35.206624 12134 tablet_replica.cc:330] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b: stopping tablet replica
I20260927 11:14:35.210289 12135 ts_tablet_manager.cc:1905] T 5235213d2edc4dc8b5bb84a830e01311 P 8a32183ac1e44e0ab839ee84b5a85976: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:14:35.210441 12134 raft_consensus.cc:2229] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:14:35.210623 12134 raft_consensus.cc:2258] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:14:35.212006 12134 ts_tablet_manager.cc:1905] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 11:14:35.212540 11593 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45331 (error 108)
I20260927 11:14:35.213733 12134 ts_tablet_manager.cc:1918] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:14:35.213807 12134 log.cc:1196] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/5235213d2edc4dc8b5bb84a830e01311
I20260927 11:14:35.213982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 11827
I20260927 11:14:35.214874 12134 ts_tablet_manager.cc:1939] T 5235213d2edc4dc8b5bb84a830e01311 P 9b881d26e126437ab1332fa6ee4a088b: Deleting consensus metadata
I20260927 11:14:35.215793 11592 catalog_manager.cc:4841] TS 9b881d26e126437ab1332fa6ee4a088b (127.0.98.3:32891): tablet 5235213d2edc4dc8b5bb84a830e01311 (table test-table [id=8e5bad07a9ed40d99d74870afa6a5e8b]) successfully deleted
I20260927 11:14:35.220628 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.220507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45331 (local address 127.0.0.1:50280)
0927 11:14:35.220552 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 11:14:35.220584 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 11:14:35.220831 11593 catalog_manager.cc:4584] TS 721e73efda5749a983cc88711b0578ac (127.0.98.1:45331): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5235213d2edc4dc8b5bb84a830e01311: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45331: connect: Connection refused (error 111)
I20260927 11:14:35.221391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 11964
I20260927 11:14:35.222040 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:14:35.221499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42155 (local address 127.0.0.1:50662)
0927 11:14:35.221976 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 11:14:35.222005 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459}
I20260927 11:14:35.226923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 11576
I20260927 11:14:35.235868 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 11:14:35.579556 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 11:14:35.579641 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 11:14:35.924583 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 11:14:35.924 UTC [11162] LOG: received smart shutdown request
2026-09-27 11:14:35.928 UTC [11162] LOG: background worker "logical replication launcher" (PID 11171) exited with exit code 1
2026-09-27 11:14:35.928 UTC [11166] LOG: shutting down
2026-09-27 11:14:36.066 UTC [11162] LOG: database system is shut down
2026-09-27T11:14:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (42741 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:14:36 dist-test-slave-9hq9 krb5kdc[12146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:14:36 dist-test-slave-9hq9 krb5kdc[12146](info): set up 2 sockets
Sep 27 11:14:36 dist-test-slave-9hq9 krb5kdc[12146](info): commencing operation
krb5kdc: starting...
W20260927 11:14:37.189782 392 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:14:37 dist-test-slave-9hq9 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:14:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:14:37Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 11:14:37.339987 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskGpH_nd/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/postgres -l logfile start
2026-09-27 11:14:39.623 UTC [12176] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 11:14:39.624 UTC [12176] LOG: listening on IPv4 address "127.0.98.20", port 53439
2026-09-27 11:14:39.627 UTC [12176] LOG: listening on Unix socket "/tmp/.s.PGSQL.53439"
2026-09-27 11:14:39.660 UTC [12179] LOG: database system was shut down at 2026-09-27 11:14:38 UTC
2026-09-27 11:14:39.664 UTC [12176] LOG: database system is ready to accept connections
I20260927 11:14:40.601639 392 mini_postgres.cc:96] Postgres bound to 53439
2026-09-27 11:14:40.607 UTC [12187] FATAL: database "slave" does not exist
127.0.98.20:53439 - accepting connections
I20260927 11:14:40.608001 392 mini_ranger.cc:161] Starting Ranger...
I20260927 11:14:40.621673 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 11:14:41.036326 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 11:14:41.036409 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-admin
2026-09-27 11:14:41,300 [I] DB FLAVOR :POSTGRES
2026-09-27 11:14:41,300 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 11:14:41,300 [I] Checking connection..
2026-09-27 11:14:41,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 11:14:41,539 [I] Checking connection passed.
2026-09-27 11:14:41,540 [I] --------- Verifying version history table ---------
2026-09-27 11:14:41,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 11:14:41,780 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 11:14:41,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 11:14:41,999 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 11:14:42,000 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 11:14:42,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 11:14:42.217 UTC [12251] WARNING: there is no transaction in progress
2026-09-27 11:14:42,223 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 11:14:42,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 11:14:42,435 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 11:14:42,435 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 11:14:42,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 11:14:42,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 11:14:42,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9hq9.c.gcp-upstream.internal','N') ;"
2026-09-27 11:14:43,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 11:14:43,316 [I] Table x_portal_user does not exist in database ranger
2026-09-27 11:14:43,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 11:14:43,534 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 11:14:43,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 11:14:43,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 11:14:44,020 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 11:14:44,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 11:14:44.884 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:44.899 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:44.915 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:44.929 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:44.944 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:44.961 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:45.023 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:45.029 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:45.043 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:45.052 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:45.061 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:45.074 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:45.083 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:45.089 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:45.095 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:45.101 UTC [12359] WARNING: there is no transaction in progress
2026-09-27 11:14:45,672 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 11:14:45,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 11:14:45,916 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 11:14:45,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 11:14:46,146 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 11:14:46,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 11:14:46,392 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 11:14:46,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 11:14:46,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 11:14:46,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9hq9.c.gcp-upstream.internal';"
2026-09-27 11:14:47,062 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 11:14:47,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 11:14:47,300 [I] DB_PATCHES have already been applied
I20260927 11:14:47.305426 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskGpH_nd/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 11:14:47.305480 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 11:14:47.308297 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:37339
Sep 27, 2026 11:14:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 11:14:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 11:14:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 11:14:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 11:14:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 11:14:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 11:14:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 11:14:48.405215 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.797s user 0.000s sys 0.008s
Sep 27, 2026 11:15:06 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 11:15:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 11:15:06 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 11:15:06 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 11:15:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 11:15:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 11:15:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 11:15:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 11:15:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 11:15:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 11:15:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 11:15:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 11:15:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 11:15:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 11:15:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 11:15:06 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 11:15:06 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 11:15:07 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 11:15:08.074820 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 11:15:08 dist-test-slave-9hq9 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507708, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 11:15:08.153479 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 11:15:08.161872 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 11:15:08.962116 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 11:15:08.962196 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-kms
I20260927 11:15:08.962213 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:53439
2026-09-27 11:15:09,121 [I] DB FLAVOR :POSTGRES
2026-09-27 11:15:09,121 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 11:15:09,121 [I] Checking connection
2026-09-27 11:15:09,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 11:15:09,391 [I] connection success
2026-09-27 11:15:09,391 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 11:15:09,391 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 11:15:09,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 11:15:09,694 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 11:15:09,694 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 11:15:09,694 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 11:15:09,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53439/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 11:15:09,978 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 11:15:10.410365 392 mini_ranger_kms.cc:310] Created kms service
I20260927 11:15:10.596721 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 11:15:10.636269 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 11:15:10 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507708, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:15:11.001871 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 11:15:11.001969 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskGpH_nd/build/release/bin/postgresql.jar:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskGpH_nd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 11:15:11.001988 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 11:15:11.005115 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12539
I20260927 11:15:11.005192 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:43041
Sep 27, 2026 11:15:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 11:15:12 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 11:15:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 11:15:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 11:15:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 11:15:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 11:15:13.242244 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.089s user 0.000s sys 0.143s
I20260927 11:15:13.242343 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 11:15:13.242401 392 mini_ranger_kms.cc:401] 127.0.98.20:43041/kms/v1/keys
Sep 27 11:15:13 dist-test-slave-9hq9 krb5kdc[12146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790507713, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 11:15:13 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507713, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 11:15:14 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507713, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 11:15:16 dist-test-slave-9hq9 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:15:16.257752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39331
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39331
--ranger_config_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43041/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:16.335464 12590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:16.335722 12590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:16.335748 12590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:16.336849 12590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:15:16.336887 12590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:16.336901 12590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:16.336913 12590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:15:16.336925 12590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:15:16.338548 12590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:43041/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39331
--ranger_config_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:16.338766 12590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:16.338979 12590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:16.343844 12595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:16.343933 12590 server_base.cc:1034] running on GCE node
W20260927 11:15:16.343961 12598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:16.343853 12596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:16.344323 12590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:16.344624 12590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:16.345770 12590 hybrid_clock.cc:648] HybridClock initialized: now 1790507716345751 us; error 36 us; skew 500 ppm
Sep 27 11:15:16 dist-test-slave-9hq9 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:16.348131 12590 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:15:16.348570 12590 webserver.cc:460] Webserver started at http://127.0.98.62:33937/ using document root <none> and password file <none>
I20260927 11:15:16.348793 12590 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:16.348829 12590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:16.348941 12590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:15:16 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:15:16.392378 12590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "09e97c5350a04ea19e44b9392ce8f974"
format_stamp: "Formatted at 2026-09-27 11:15:16 on dist-test-slave-9hq9"
server_key: "37ff5f31318610fffd87f070bf0e19bb"
server_key_iv: "092d3b7ab343cfc8007d69453e6ddb62"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:16.392794 12590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "09e97c5350a04ea19e44b9392ce8f974"
format_stamp: "Formatted at 2026-09-27 11:15:16 on dist-test-slave-9hq9"
server_key: "37ff5f31318610fffd87f070bf0e19bb"
server_key_iv: "092d3b7ab343cfc8007d69453e6ddb62"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:16.394315 12590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260927 11:15:16.405166 12605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:16.405407 12590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:15:16.405516 12590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "09e97c5350a04ea19e44b9392ce8f974"
format_stamp: "Formatted at 2026-09-27 11:15:16 on dist-test-slave-9hq9"
server_key: "37ff5f31318610fffd87f070bf0e19bb"
server_key_iv: "092d3b7ab343cfc8007d69453e6ddb62"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:16.405594 12590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:16.426632 12590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:16.428735 12590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:16.428889 12590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:16.433944 12590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39331
I20260927 11:15:16.434005 12657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39331 every 8 connection(s)
I20260927 11:15:16.434401 12590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:15:16.435114 12658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:16.437291 12658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974: Bootstrap starting.
I20260927 11:15:16.437855 12658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:16.438228 12658 log.cc:824] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:16.438897 12658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974: No bootstrap required, opened a new log
I20260927 11:15:16.440853 12658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e97c5350a04ea19e44b9392ce8f974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } }
I20260927 11:15:16.441016 12658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:16.441071 12658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e97c5350a04ea19e44b9392ce8f974, State: Initialized, Role: FOLLOWER
I20260927 11:15:16.441232 12658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e97c5350a04ea19e44b9392ce8f974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } }
I20260927 11:15:16.441331 12658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:15:16.441375 12658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:15:16.441422 12658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:16.441917 12658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e97c5350a04ea19e44b9392ce8f974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } }
I20260927 11:15:16.442029 12658 leader_election.cc:302] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09e97c5350a04ea19e44b9392ce8f974; no voters:
I20260927 11:15:16.442225 12658 leader_election.cc:288] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:15:16.442246 12661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:15:16.442420 12661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 1 LEADER]: Becoming Leader. State: Replica: 09e97c5350a04ea19e44b9392ce8f974, State: Running, Role: LEADER
I20260927 11:15:16.442582 12661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e97c5350a04ea19e44b9392ce8f974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } }
I20260927 11:15:16.442610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 12590
I20260927 11:15:16.442701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:15:16.442898 392 external_mini_cluster.cc:1428] Setting key 1dd5751b1bac3ad5d7adda5a95243391
I20260927 11:15:16.442931 12661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09e97c5350a04ea19e44b9392ce8f974. Latest consensus state: current_term: 1 leader_uuid: "09e97c5350a04ea19e44b9392ce8f974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e97c5350a04ea19e44b9392ce8f974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } } }
I20260927 11:15:16.443008 12661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:16.442943 12662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09e97c5350a04ea19e44b9392ce8f974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e97c5350a04ea19e44b9392ce8f974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } } }
I20260927 11:15:16.443059 12662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:16.443809 12658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:15:16.443943 12665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:15:16.444489 12665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:15:16.445292 12658 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 11:15:16.450400 12665 catalog_manager.cc:1347] Generated new cluster ID: 5a3c02b73f0c4a4ab1ca9f0a29eb61c8
I20260927 11:15:16.450495 12665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:15:16 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:16.456408 12667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:16.443560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.0.98.62:39331)
0927 11:15:16.443885 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:16.443895 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:16.443927 (+ 32us) server_negotiation.cc:407] Connection header received
0927 11:15:16.444047 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:16.444054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:16.444108 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:16.444247 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:16.445210 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:16.446359 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:16.448134 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:16.449819 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:16.449971 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:16.452480 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:16.452484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:16.452484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:16.452501 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:16.454796 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:16.455162 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:16.455165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:16.455165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:16.455265 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:16.455487 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:16.455488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:16.455488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:16.455739 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:16.455938 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:16.456010 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 11:15:16.456175 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":68,"threads_started":1}
I20260927 11:15:16.466809 12665 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:15:16.467535 12665 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:15:16.479657 12665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974: Generated new TSK 0
I20260927 11:15:16.479902 12665 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:15:17.140112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39331
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:17.223394 12703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:17.223610 12703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:17.223644 12703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:17.225306 12703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:17.225375 12703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:17.225414 12703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:15:17.227451 12703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:43041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:17.227661 12703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:17.227921 12703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:17.233726 12708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:17.233870 12711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:17.233747 12709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:17.235170 12703 server_base.cc:1034] running on GCE node
I20260927 11:15:17.235388 12703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:17.235622 12703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:17.236791 12703 hybrid_clock.cc:648] HybridClock initialized: now 1790507717236775 us; error 31 us; skew 500 ppm
Sep 27 11:15:17 dist-test-slave-9hq9 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:17.239424 12703 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:15:17.239881 12703 webserver.cc:460] Webserver started at http://127.0.98.1:35637/ using document root <none> and password file <none>
I20260927 11:15:17.240108 12703 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:17.240151 12703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:17.240267 12703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:15:17 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:15:17.257254 12703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "16c57d96d8364c4e9d5c58b03edb1979"
format_stamp: "Formatted at 2026-09-27 11:15:17 on dist-test-slave-9hq9"
server_key: "986a519f546aacc53a8d2fa246c0b648"
server_key_iv: "980258a62164751909896320e6f342b3"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:17.257696 12703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "16c57d96d8364c4e9d5c58b03edb1979"
format_stamp: "Formatted at 2026-09-27 11:15:17 on dist-test-slave-9hq9"
server_key: "986a519f546aacc53a8d2fa246c0b648"
server_key_iv: "980258a62164751909896320e6f342b3"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:17.261672 12703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 11:15:17.273731 12718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:17.273900 12703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:15:17.273993 12703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "16c57d96d8364c4e9d5c58b03edb1979"
format_stamp: "Formatted at 2026-09-27 11:15:17 on dist-test-slave-9hq9"
server_key: "986a519f546aacc53a8d2fa246c0b648"
server_key_iv: "980258a62164751909896320e6f342b3"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:17.274087 12703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:17.287865 12703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:17.290436 12703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:17.290603 12703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:17.290853 12703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:17.291201 12703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:15:17.291270 12703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:17.291307 12703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:15:17.291339 12703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:17.297407 12703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33583
I20260927 11:15:17.297865 12703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:15:17.298799 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 12703
I20260927 11:15:17.298895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:15:17.299103 392 external_mini_cluster.cc:1428] Setting key b2407bb57e4086ef10a705886cea9c62
I20260927 11:15:17.301728 12833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33583 every 8 connection(s)
Sep 27 11:15:17 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 11:15:17.315860 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:17.302345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39331 (local address 127.0.98.1:52201)
0927 11:15:17.305057 (+ 2712us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:17.305103 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:17.305713 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:17.306151 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:17.306155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:17.306389 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:17.306642 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:17.306647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:17.307703 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:17.307704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:17.309045 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:17.309047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:17.309665 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:17.309666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:17.309837 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:17.310466 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:17.310475 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:17.312167 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:17.314315 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:17.314331 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:17.314333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:17.314597 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:17.314812 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:17.314813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:17.314813 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:17.314930 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:17.315205 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:17.315210 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:17.315401 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 11:15:17.315509 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 11:15:17.315647 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2670,"thread_start_us":92,"threads_started":1}
I20260927 11:15:17.316381 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:17.302614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52201 (local address 127.0.98.62:39331)
0927 11:15:17.305900 (+ 3286us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:17.305902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:17.305911 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:15:17.305933 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:17.305934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:17.305945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:17.306011 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:17.306748 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:17.307599 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:17.309141 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:17.309489 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:17.309585 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:17.312269 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:17.312270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:17.312270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:17.312277 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:17.314227 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:17.314681 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:17.314682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:17.314682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:17.314746 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:17.314999 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:17.314999 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:17.315000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:17.315125 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:17.315181 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:17.316185 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0927 11:15:17.316267 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3254,"thread_start_us":70,"threads_started":1}
I20260927 11:15:17.317019 12834 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39331
I20260927 11:15:17.317101 12834 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:17.317262 12834 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
I20260927 11:15:17.317821 12622 ts_manager.cc:194] Registered new tserver with Master: 16c57d96d8364c4e9d5c58b03edb1979 (127.0.98.1:33583)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 11:15:17.318823 12622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:15:17.334852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39331
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:17.447685 12840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:17.447961 12840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:17.447993 12840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:17.449589 12840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:17.449646 12840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:17.449682 12840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:15:17.451716 12840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:43041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:17.451956 12840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:17.452248 12840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:17.458070 12848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:17.458094 12847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:17.458031 12850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:17.458736 12840 server_base.cc:1034] running on GCE node
I20260927 11:15:17.458949 12840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:17.459210 12840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:17.460405 12840 hybrid_clock.cc:648] HybridClock initialized: now 1790507717460383 us; error 35 us; skew 500 ppm
Sep 27 11:15:17 dist-test-slave-9hq9 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:17.463655 12840 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:15:17.464145 12840 webserver.cc:460] Webserver started at http://127.0.98.2:41047/ using document root <none> and password file <none>
I20260927 11:15:17.464413 12840 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:17.464511 12840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:17.464660 12840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:15:17 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:15:17.477775 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "3e79c692c9cd4d5aae629e060cf997e1"
format_stamp: "Formatted at 2026-09-27 11:15:17 on dist-test-slave-9hq9"
server_key: "0986ae6f1fb7897d4af446919ef11fde"
server_key_iv: "ccdadefb4aac79089071156b083477fb"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:17.478349 12840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3e79c692c9cd4d5aae629e060cf997e1"
format_stamp: "Formatted at 2026-09-27 11:15:17 on dist-test-slave-9hq9"
server_key: "0986ae6f1fb7897d4af446919ef11fde"
server_key_iv: "ccdadefb4aac79089071156b083477fb"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:17.485746 12840 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
Sep 27 11:15:17 dist-test-slave-9hq9 krb5kdc[12146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790507717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:17.498884 12857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:17.499073 12840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:15:17.499148 12840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "3e79c692c9cd4d5aae629e060cf997e1"
format_stamp: "Formatted at 2026-09-27 11:15:17 on dist-test-slave-9hq9"
server_key: "0986ae6f1fb7897d4af446919ef11fde"
server_key_iv: "ccdadefb4aac79089071156b083477fb"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:17.499225 12840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:17.526109 12840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:17.528388 12840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:17.528568 12840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:17.528802 12840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:17.529230 12840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:15:17.529312 12840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:17.529385 12840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:15:17.529438 12840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:17.536209 12840 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34445
I20260927 11:15:17.536301 12970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34445 every 8 connection(s)
I20260927 11:15:17.536609 12840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:15:17.542234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 12840
I20260927 11:15:17.542331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
Sep 27 11:15:17 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:17.542524 392 external_mini_cluster.cc:1428] Setting key 23ac8445359da35760de6cbbb4db35f4
I20260927 11:15:17.546938 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:17.537285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34679 (local address 127.0.98.62:39331)
0927 11:15:17.537349 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:17.537351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:17.537531 (+ 180us) server_negotiation.cc:407] Connection header received
0927 11:15:17.538010 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:17.538011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:17.538022 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:17.538097 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:17.538681 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:17.539451 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:17.540353 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:17.540651 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:17.540712 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:17.543800 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:17.543801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:17.543802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:17.543809 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:17.545509 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:17.545947 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:17.545948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:17.545949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:17.546000 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:17.546274 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:17.546275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:17.546275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:17.546392 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:17.546431 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:17.546754 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 11:15:17.546845 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:15:17.547355 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:17.537241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39331 (local address 127.0.98.2:34679)
0927 11:15:17.537409 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:17.537447 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:17.537917 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:17.538185 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:17.538189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:17.538424 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:17.538605 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:17.538610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:17.539531 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:17.539532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:17.540279 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:17.540281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:17.540730 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:17.540731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:17.540818 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:17.541285 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:17.541293 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:17.543317 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:17.545594 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:17.545599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:17.545601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:17.545868 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:17.546098 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:17.546098 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:17.546099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:17.546207 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:17.546472 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:17.546476 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:17.546623 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 11:15:17.547020 (+ 397us) client_negotiation.cc:240] Negotiation successful
0927 11:15:17.547154 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":102,"threads_started":1}
I20260927 11:15:17.547962 12971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39331
I20260927 11:15:17.548048 12971 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:17.548240 12971 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
I20260927 11:15:17.548646 12622 ts_manager.cc:194] Registered new tserver with Master: 3e79c692c9cd4d5aae629e060cf997e1 (127.0.98.2:34445)
I20260927 11:15:17.549161 12622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34679
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:15:17.568297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39331
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:17.654186 12977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:17.654387 12977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:17.654420 12977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:17.655967 12977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:17.656056 12977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:17.656113 12977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:15:17.658140 12977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:43041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:17.658466 12977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:17.658707 12977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:17.664136 12982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:17.665091 12985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:17.665792 12983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:17.666106 12977 server_base.cc:1034] running on GCE node
I20260927 11:15:17.666446 12977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:17.666697 12977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:17.667863 12977 hybrid_clock.cc:648] HybridClock initialized: now 1790507717667843 us; error 31 us; skew 500 ppm
Sep 27 11:15:17 dist-test-slave-9hq9 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:17.670295 12977 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:15:17.670753 12977 webserver.cc:460] Webserver started at http://127.0.98.3:40905/ using document root <none> and password file <none>
I20260927 11:15:17.670975 12977 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:17.671042 12977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:17.671164 12977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 11:15:17 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:15:17.686105 12977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "a46e12690d5a41f987be0db19d40ccdd"
format_stamp: "Formatted at 2026-09-27 11:15:17 on dist-test-slave-9hq9"
server_key: "6c38077b15b14cb10af07b15018b0ade"
server_key_iv: "2ff7da7474478caaed8ba62777ad4dd4"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:17.686599 12977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a46e12690d5a41f987be0db19d40ccdd"
format_stamp: "Formatted at 2026-09-27 11:15:17 on dist-test-slave-9hq9"
server_key: "6c38077b15b14cb10af07b15018b0ade"
server_key_iv: "2ff7da7474478caaed8ba62777ad4dd4"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:17.690485 12977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
I20260927 11:15:17.706141 12992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:17.706362 12977 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 11:15:17.706452 12977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "a46e12690d5a41f987be0db19d40ccdd"
format_stamp: "Formatted at 2026-09-27 11:15:17 on dist-test-slave-9hq9"
server_key: "6c38077b15b14cb10af07b15018b0ade"
server_key_iv: "2ff7da7474478caaed8ba62777ad4dd4"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:17.706530 12977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:17.740341 12977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:17.742813 12977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:17.743002 12977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:17.743371 12977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:17.743834 12977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:15:17.743921 12977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:17.744081 12977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:15:17.744130 12977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:17.750660 12977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39235
I20260927 11:15:17.751078 12977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:15:17.754566 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 12977
I20260927 11:15:17.754658 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:15:17.754855 392 external_mini_cluster.cc:1428] Setting key 46122d513f9b669b20da513f2ba120f4
I20260927 11:15:17.761242 13105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39235 every 8 connection(s)
Sep 27 11:15:17 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:17.769845 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:17.761854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38869 (local address 127.0.98.62:39331)
0927 11:15:17.761893 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:17.761895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:17.761903 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:15:17.762448 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:17.762449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:17.762460 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:17.762505 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:17.763031 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:17.763545 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:17.764540 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:17.764740 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:17.764827 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:17.767032 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:17.767033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:17.767034 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:17.767041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:17.768471 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:17.768881 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:17.768882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:17.768883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:17.768946 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:17.769199 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:17.769200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:17.769200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:17.769314 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:17.769383 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:17.769674 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 11:15:17.769749 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:15:17.770027 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:17.761565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39331 (local address 127.0.98.3:38869)
0927 11:15:17.761730 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:17.761770 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:17.762347 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:17.762578 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:17.762581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:17.762752 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:17.762938 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:17.762942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:17.763636 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:17.763638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:17.764439 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:17.764441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:17.764847 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:17.764848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:17.764957 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:17.765469 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:17.765477 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:17.766946 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:17.768569 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:17.768575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:17.768577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:17.768794 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:17.769011 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:17.769011 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:17.769012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:17.769124 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:17.769385 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:17.769392 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:17.769542 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 11:15:17.769654 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 11:15:17.769789 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
I20260927 11:15:17.770563 13106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39331
I20260927 11:15:17.770653 13106 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:17.770823 13106 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
I20260927 11:15:17.771159 12622 ts_manager.cc:194] Registered new tserver with Master: a46e12690d5a41f987be0db19d40ccdd (127.0.98.3:39235)
I20260927 11:15:17.771516 12622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38869
I20260927 11:15:17.777053 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:15:17.777710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 12703
I20260927 11:15:17.782531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 12840
I20260927 11:15:17.786527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 12977
I20260927 11:15:17.791463 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 12590
I20260927 11:15:17.797504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39331
--webserver_interface=127.0.98.62
--webserver_port=33937
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39331
--ranger_config_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43041/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43041/kms with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:17.874964 13109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:17.875133 13109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:17.875152 13109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:17.876349 13109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:15:17.876389 13109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:17.876402 13109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:17.876446 13109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:15:17.876470 13109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:15:17.877946 13109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43041/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39331
--ranger_config_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:17.878180 13109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:17.878365 13109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:17.882716 13117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:17.882728 13114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:17.882753 13109 server_base.cc:1034] running on GCE node
W20260927 11:15:17.882846 13115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:17.883114 13109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:17.883405 13109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:17.884562 13109 hybrid_clock.cc:648] HybridClock initialized: now 1790507717884551 us; error 27 us; skew 500 ppm
Sep 27 11:15:17 dist-test-slave-9hq9 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:17.886930 13109 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:15:17.887341 13109 webserver.cc:460] Webserver started at http://127.0.98.62:33937/ using document root <none> and password file <none>
I20260927 11:15:17.887532 13109 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:17.887579 13109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 11:15:17 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:15:17.946000 13109 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:15:17.946681 13125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:17.946856 13109 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:17.946964 13109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "09e97c5350a04ea19e44b9392ce8f974"
format_stamp: "Formatted at 2026-09-27 11:15:16 on dist-test-slave-9hq9"
server_key: "37ff5f31318610fffd87f070bf0e19bb"
server_key_iv: "092d3b7ab343cfc8007d69453e6ddb62"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:17.947321 13109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:17.956187 13109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:17.958168 13109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:17.958314 13109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:17.962214 13109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39331
I20260927 11:15:17.962255 13177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39331 every 8 connection(s)
I20260927 11:15:17.962647 13109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:15:17.963694 13178 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:15:17.964322 13178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974: Bootstrap starting.
I20260927 11:15:17.971706 13178 log.cc:824] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:17.972747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 13109
I20260927 11:15:17.973009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33583
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39331
--webserver_port=35637
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43041/kms with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:15:17.973368 13178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:15:17.973626 13178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974: Bootstrap complete.
I20260927 11:15:17.975908 13178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e97c5350a04ea19e44b9392ce8f974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } }
I20260927 11:15:17.976150 13178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e97c5350a04ea19e44b9392ce8f974, State: Initialized, Role: FOLLOWER
I20260927 11:15:17.976337 13178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e97c5350a04ea19e44b9392ce8f974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } }
I20260927 11:15:17.976428 13178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:15:17.976483 13178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:15:17.976532 13178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 1 FOLLOWER]: Advancing to term 2
I20260927 11:15:17.977466 13178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e97c5350a04ea19e44b9392ce8f974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } }
I20260927 11:15:17.977579 13178 leader_election.cc:302] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09e97c5350a04ea19e44b9392ce8f974; no voters:
I20260927 11:15:17.977767 13178 leader_election.cc:288] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 11:15:17.977806 13182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 11:15:17.977910 13182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [term 2 LEADER]: Becoming Leader. State: Replica: 09e97c5350a04ea19e44b9392ce8f974, State: Running, Role: LEADER
I20260927 11:15:17.978022 13178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:15:17.978084 13182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e97c5350a04ea19e44b9392ce8f974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } }
I20260927 11:15:17.978418 13184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "09e97c5350a04ea19e44b9392ce8f974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e97c5350a04ea19e44b9392ce8f974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } } }
I20260927 11:15:17.978513 13184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:17.978813 13183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09e97c5350a04ea19e44b9392ce8f974. Latest consensus state: current_term: 2 leader_uuid: "09e97c5350a04ea19e44b9392ce8f974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e97c5350a04ea19e44b9392ce8f974" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39331 } } }
I20260927 11:15:17.978905 13183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974 [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:17.979133 13190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:15:17.979555 13178 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 11:15:17.979655 13190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:15:17.979857 13190 catalog_manager.cc:1259] Loaded cluster ID: 5a3c02b73f0c4a4ab1ca9f0a29eb61c8
I20260927 11:15:17.979933 13190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:15:17.984018 13190 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:15:17.984351 13190 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974: Loaded TSK: 0
I20260927 11:15:17.984686 13190 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 11:15:18.083932 13181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:18.084280 13181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:18.084372 13181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:18.085965 13181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:18.086078 13181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:18.086162 13181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:15:18.088207 13181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35637
--tserver_master_addrs=127.0.98.62:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:18.088419 13181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:18.088702 13181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:18.094841 13219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:18.095007 13217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:18.095036 13181 server_base.cc:1034] running on GCE node
W20260927 11:15:18.095438 13216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:18.095685 13181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:18.095938 13181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:18.097114 13181 hybrid_clock.cc:648] HybridClock initialized: now 1790507718097102 us; error 30 us; skew 500 ppm
Sep 27 11:15:18 dist-test-slave-9hq9 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:18.100143 13181 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:15:18.100710 13181 webserver.cc:460] Webserver started at http://127.0.98.1:35637/ using document root <none> and password file <none>
I20260927 11:15:18.100994 13181 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:18.101061 13181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 11:15:18 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:15:18.127178 13181 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:15:18.127959 13227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:18.128137 13181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:15:18.128230 13181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "16c57d96d8364c4e9d5c58b03edb1979"
format_stamp: "Formatted at 2026-09-27 11:15:17 on dist-test-slave-9hq9"
server_key: "986a519f546aacc53a8d2fa246c0b648"
server_key_iv: "980258a62164751909896320e6f342b3"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:18.128648 13181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:18.146397 13181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:18.152828 13181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:18.153033 13181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:18.153276 13181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:18.153649 13181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:15:18.153713 13181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:18.153748 13181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:15:18.153808 13181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:18.160061 13181 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33583
I20260927 11:15:18.160503 13181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:15:18.162472 13340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33583 every 8 connection(s)
I20260927 11:15:18.164623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 13181
I20260927 11:15:18.164880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34445
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39331
--webserver_port=41047
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43041/kms with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 11:15:18 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:18.183840 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:18.163343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39331 (local address 127.0.98.1:54549)
0927 11:15:18.163700 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:18.163743 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:18.164287 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:18.166920 (+ 2633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:18.166924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:18.167145 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:18.167408 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:18.167415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:18.169134 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:18.169135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:18.170536 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:18.170539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:18.171435 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:18.171436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:18.171579 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:18.172129 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:18.172139 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:18.173804 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:18.182177 (+ 8373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:18.182183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:18.182185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:18.182447 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:18.182696 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:18.182697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:18.182697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:18.182800 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:18.183254 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:18.183259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:18.183403 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 11:15:18.183485 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 11:15:18.183623 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":81,"threads_started":1}
I20260927 11:15:18.184746 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:18.163378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54549 (local address 127.0.98.62:39331)
0927 11:15:18.166447 (+ 3069us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:18.166456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:18.166490 (+ 34us) server_negotiation.cc:407] Connection header received
0927 11:15:18.166624 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:18.166631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:18.166704 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:18.167544 (+ 840us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:18.167632 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:18.169020 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:18.170649 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:18.171307 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:18.171421 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:18.173924 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:18.173928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:18.173929 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:18.173940 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:18.182083 (+ 8143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:18.182532 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:18.182535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:18.182535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:18.182628 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:18.182864 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:18.182864 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:18.182865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:18.183173 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:18.184387 (+ 1214us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:18.184458 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 11:15:18.184565 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3006,"thread_start_us":64,"threads_started":1}
I20260927 11:15:18.185622 13341 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39331
I20260927 11:15:18.185707 13341 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:18.185895 13341 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
W20260927 11:15:18.186786 13341 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39331 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 11:15:18.186857 13341 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:18.186918 13341 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
I20260927 11:15:18.187299 13341 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:18.187372 13341 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
W20260927 11:15:18.187731 13341 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 11:15:18.268131 13345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:18.268393 13345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:18.268457 13345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:18.269757 13345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:18.269825 13345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:18.269868 13345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:15:18.271469 13345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41047
--tserver_master_addrs=127.0.98.62:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:18.271710 13345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:18.271984 13345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:18.277349 13350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:18.277293 13353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:18.277336 13351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:18.278213 13345 server_base.cc:1034] running on GCE node
I20260927 11:15:18.278409 13345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:18.278671 13345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:18.279834 13345 hybrid_clock.cc:648] HybridClock initialized: now 1790507718279817 us; error 30 us; skew 500 ppm
Sep 27 11:15:18 dist-test-slave-9hq9 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:18.282720 13345 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:15:18.283202 13345 webserver.cc:460] Webserver started at http://127.0.98.2:41047/ using document root <none> and password file <none>
I20260927 11:15:18.283423 13345 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:18.283500 13345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 11:15:18 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:15:18.311944 13345 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260927 11:15:18.312686 13361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:18.312852 13345 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:15:18.312944 13345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "3e79c692c9cd4d5aae629e060cf997e1"
format_stamp: "Formatted at 2026-09-27 11:15:17 on dist-test-slave-9hq9"
server_key: "0986ae6f1fb7897d4af446919ef11fde"
server_key_iv: "ccdadefb4aac79089071156b083477fb"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:18.313293 13345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:18.343322 13345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:18.345808 13345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:18.345955 13345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:18.346251 13345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:18.346616 13345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:15:18.346678 13345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:18.346760 13345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:15:18.346792 13345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:18.353965 13345 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34445
I20260927 11:15:18.354085 13474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34445 every 8 connection(s)
I20260927 11:15:18.354422 13345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:15:18 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:18.363277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 13345
I20260927 11:15:18.363584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:43041/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39235
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39331
--webserver_port=40905
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43041/kms with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:15:18.366051 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:18.355239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46059 (local address 127.0.98.62:39331)
0927 11:15:18.355287 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:18.355289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:18.355531 (+ 242us) server_negotiation.cc:407] Connection header received
0927 11:15:18.356078 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:18.356079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:18.356094 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:18.356145 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:18.356671 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:18.357199 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:18.358207 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:18.358416 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:18.358501 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:18.360694 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:18.360695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:18.360695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:18.360703 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:18.362456 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:18.362928 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:18.362929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:18.362930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:18.362993 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:18.365258 (+ 2265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:18.365259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:18.365260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:18.365391 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:18.365476 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:18.365876 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 11:15:18.365957 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:15:18.366550 13477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:18.355244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39331 (local address 127.0.98.2:46059)
0927 11:15:18.355413 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:18.355448 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:18.355962 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:18.356218 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:18.356221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:18.356395 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:18.356577 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:18.356582 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:18.357294 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:18.357295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:18.358128 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:18.358130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:18.358521 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:18.358522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:18.358632 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:18.359129 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:18.359137 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:18.360591 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:18.362553 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:18.362559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:18.362561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:18.362842 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:18.365034 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:18.365035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:18.365036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:18.365170 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:18.365530 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:18.365536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:18.365706 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 11:15:18.366167 (+ 461us) client_negotiation.cc:240] Negotiation successful
0927 11:15:18.366319 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260927 11:15:18.370241 13475 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39331
I20260927 11:15:18.370337 13475 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:18.370510 13475 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
W20260927 11:15:18.371181 13475 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39331 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 11:15:18.371249 13475 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:18.371307 13475 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
I20260927 11:15:18.371671 13475 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:18.371743 13475 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
W20260927 11:15:18.372052 13475 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 11:15:18.487380 13478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:18.487648 13478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:18.487689 13478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:18.489449 13478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:18.489511 13478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:18.489548 13478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:15:18.491674 13478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:43041/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40905
--tserver_master_addrs=127.0.98.62:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:18.491904 13478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:18.492170 13478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:18.497376 13487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:18.497416 13485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:18.497524 13478 server_base.cc:1034] running on GCE node
W20260927 11:15:18.497598 13484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:18.497798 13478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:18.498042 13478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:18.499228 13478 hybrid_clock.cc:648] HybridClock initialized: now 1790507718499213 us; error 24 us; skew 500 ppm
Sep 27 11:15:18 dist-test-slave-9hq9 krb5kdc[12146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:18.501945 13478 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:15:18.502441 13478 webserver.cc:460] Webserver started at http://127.0.98.3:40905/ using document root <none> and password file <none>
I20260927 11:15:18.502655 13478 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:18.502719 13478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 11:15:18 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:15:18.519151 13478 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:15:18.519794 13495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:18.520057 13478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:15:18.520174 13478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "a46e12690d5a41f987be0db19d40ccdd"
format_stamp: "Formatted at 2026-09-27 11:15:17 on dist-test-slave-9hq9"
server_key: "6c38077b15b14cb10af07b15018b0ade"
server_key_iv: "2ff7da7474478caaed8ba62777ad4dd4"
server_key_version: "kuduclusterkey@0"
I20260927 11:15:18.520618 13478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:18.551635 13478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:18.553728 13478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:18.553889 13478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:18.554136 13478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:18.554502 13478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:15:18.554555 13478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:18.554596 13478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:15:18.554622 13478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:18.561874 13478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39235
I20260927 11:15:18.562381 13478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:15:18.563112 13608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39235 every 8 connection(s)
I20260927 11:15:18.563921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 13478
Sep 27 11:15:18 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:18.573958 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:18.563213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34179 (local address 127.0.98.62:39331)
0927 11:15:18.563269 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:18.563272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:18.563817 (+ 545us) server_negotiation.cc:407] Connection header received
0927 11:15:18.564604 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:18.564605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:18.564616 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:18.564645 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:18.565451 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:18.566223 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:18.567271 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:18.567546 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:18.567632 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:18.570036 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:18.570037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:18.570037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:18.570044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:18.571894 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:18.572383 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:18.572384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:18.572384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:18.572436 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:18.573238 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:18.573239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:18.573239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:18.573378 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:18.573462 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:18.573796 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 11:15:18.573870 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:15:18.574147 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:18.563220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39331 (local address 127.0.98.3:34179)
0927 11:15:18.563680 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:18.563732 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:18.564490 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:18.564828 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:18.564831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:18.565049 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:18.565328 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:18.565333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:18.566309 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:18.566310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:18.567126 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:18.567128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:18.567652 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:18.567653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:18.567824 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:18.568358 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:18.568367 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:18.569910 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:18.571990 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:18.571996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:18.571998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:18.572302 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:18.572740 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:18.572741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:18.572741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:18.572884 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:18.573466 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:18.573472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:18.573639 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 11:15:18.573760 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 11:15:18.573906 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":84,"threads_started":1}
I20260927 11:15:18.574726 13609 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39331
I20260927 11:15:18.574818 13609 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:18.574988 13609 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
W20260927 11:15:18.575614 13609 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39331 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 11:15:18.575726 13609 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:18.575783 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:18.564294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.0.98.62:39331)
0927 11:15:18.565996 (+ 1702us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:18.565998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:18.567672 (+ 1674us) server_negotiation.cc:407] Connection header received
0927 11:15:18.567690 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:18.567691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:18.567701 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:18.567773 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:18.569012 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:18.569479 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:18.570506 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:18.570803 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:18.570890 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:18.572230 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:18.572232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:18.572232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:18.572239 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:18.573799 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:18.574407 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:18.574408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:18.574408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:18.574474 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:18.575217 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:18.575218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:18.575218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:18.575349 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:18.575515 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:18.575612 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 11:15:18.575691 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1684,"thread_start_us":69,"threads_started":1}
I20260927 11:15:18.576021 13609 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
I20260927 11:15:18.576655 13609 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:18.576736 13609 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
W20260927 11:15:18.577059 13609 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 11:15:19 dist-test-slave-9hq9 krb5kdc[12146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790507719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:19.187935 13341 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:19.188083 13341 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
I20260927 11:15:19.188462 13142 ts_manager.cc:194] Registered new tserver with Master: 16c57d96d8364c4e9d5c58b03edb1979 (127.0.98.1:33583)
I20260927 11:15:19.189502 13142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54549
I20260927 11:15:19.372200 13475 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:19.372354 13475 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
I20260927 11:15:19.372702 13142 ts_manager.cc:194] Registered new tserver with Master: 3e79c692c9cd4d5aae629e060cf997e1 (127.0.98.2:34445)
I20260927 11:15:19.373226 13142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46059
Sep 27 11:15:19 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:15:19.577244 13609 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:19.577382 13609 heartbeater.cc:511] Master 127.0.98.62:39331 requested a full tablet report, sending...
I20260927 11:15:19.577636 13142 ts_manager.cc:194] Registered new tserver with Master: a46e12690d5a41f987be0db19d40ccdd (127.0.98.3:39235)
I20260927 11:15:19.578183 13142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34179
I20260927 11:15:19.586171 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:15:19.596575 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:19.587170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33662 (local address 127.0.98.62:39331)
0927 11:15:19.587407 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:19.587409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:19.587569 (+ 160us) server_negotiation.cc:407] Connection header received
0927 11:15:19.587592 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:19.587592 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:19.587604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:19.587659 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:19.588343 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.589075 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.590165 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.590455 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.591683 (+ 1228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:19.593201 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:19.593202 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:19.593202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:19.593209 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:19.595082 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:19.595480 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:19.595481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:19.595482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:19.595538 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:19.595738 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:19.595739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:19.595739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:19.595871 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:19.595967 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:19.596395 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 11:15:19.596479 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":70,"threads_started":1}
I20260927 11:15:19.598697 13142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:15:19.600993 13142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:15:19.615496 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:19.610369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33583 (local address 127.0.0.1:36078)
0927 11:15:19.610579 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:19.610589 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:19.610657 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:19.611321 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:19.611324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:19.611326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:19.611387 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:19.611389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.612586 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.612587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:19.613822 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:19.613825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.615212 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.615213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:19.615292 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:19.615305 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 11:15:19.615371 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 11:15:19.615417 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":62,"threads_started":1}
I20260927 11:15:19.616068 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:19.611013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.0.98.1:33583)
0927 11:15:19.611149 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:19.611154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:19.611164 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:15:19.611197 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:19.611199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:19.611214 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:19.611450 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:19.611511 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.612481 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.613919 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.615018 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.615093 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:19.615176 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:19.615929 (+ 753us) server_negotiation.cc:299] Negotiation successful
0927 11:15:19.615982 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260927 11:15:19.616874 13275 tablet_service.cc:1467] Processing CreateTablet for tablet 8c0e6827f2834d889824e65b185f7a7e (DEFAULT_TABLE table=test-table [id=a47438e214ab44579d6b53fce852a41d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:15:19.617221 13275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c0e6827f2834d889824e65b185f7a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:19.618306 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:19.610146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39235 (local address 127.0.0.1:52132)
0927 11:15:19.610492 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:19.610745 (+ 253us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:19.610774 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:19.612811 (+ 2037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:19.612811 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:19.612814 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:19.612855 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:19.612856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.614581 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.614581 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:19.615872 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:19.615876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.618079 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.618080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:19.618135 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:19.618139 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:15:19.618184 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 11:15:19.618236 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":71,"threads_started":1}
I20260927 11:15:19.618690 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:19.611285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.0.98.3:39235)
0927 11:15:19.612616 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:19.612621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:19.612630 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:15:19.612690 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:19.612692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:19.612707 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:19.612919 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:19.612982 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.614374 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.617294 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.617955 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.618460 (+ 505us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:19.618541 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:19.618567 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 11:15:19.618606 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1299,"thread_start_us":52,"threads_started":1}
I20260927 11:15:19.619329 13543 tablet_service.cc:1467] Processing CreateTablet for tablet 8c0e6827f2834d889824e65b185f7a7e (DEFAULT_TABLE table=test-table [id=a47438e214ab44579d6b53fce852a41d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:15:19.619656 13543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c0e6827f2834d889824e65b185f7a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:19.619828 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:19.610795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34445 (local address 127.0.0.1:44046)
0927 11:15:19.610932 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:19.610941 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:19.610975 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:19.614440 (+ 3465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:19.614441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:19.614443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:19.614480 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:19.614482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.616007 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.616008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:19.617158 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:19.617160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.619601 (+ 2441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.619602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:19.619671 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:19.619675 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:15:19.619706 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 11:15:19.619740 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260927 11:15:19.620194 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:19.611408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44046 (local address 127.0.98.2:34445)
0927 11:15:19.614275 (+ 2867us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:19.614279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:19.614289 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:15:19.614322 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:19.614324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:19.614338 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:19.614426 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:19.614913 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.615942 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.618783 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.619500 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.619987 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:19.620063 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:19.620087 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 11:15:19.620124 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2834,"thread_start_us":59,"threads_started":1}
I20260927 11:15:19.620807 13409 tablet_service.cc:1467] Processing CreateTablet for tablet 8c0e6827f2834d889824e65b185f7a7e (DEFAULT_TABLE table=test-table [id=a47438e214ab44579d6b53fce852a41d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:15:19.620934 13643 tablet_bootstrap.cc:492] T 8c0e6827f2834d889824e65b185f7a7e P 16c57d96d8364c4e9d5c58b03edb1979: Bootstrap starting.
I20260927 11:15:19.621122 13409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c0e6827f2834d889824e65b185f7a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:19.621722 13643 tablet_bootstrap.cc:654] T 8c0e6827f2834d889824e65b185f7a7e P 16c57d96d8364c4e9d5c58b03edb1979: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:19.622248 13643 log.cc:824] T 8c0e6827f2834d889824e65b185f7a7e P 16c57d96d8364c4e9d5c58b03edb1979: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:19.623035 13643 tablet_bootstrap.cc:492] T 8c0e6827f2834d889824e65b185f7a7e P 16c57d96d8364c4e9d5c58b03edb1979: No bootstrap required, opened a new log
I20260927 11:15:19.623096 13643 ts_tablet_manager.cc:1397] T 8c0e6827f2834d889824e65b185f7a7e P 16c57d96d8364c4e9d5c58b03edb1979: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
Sep 27 11:15:19 dist-test-slave-9hq9 krb5kdc[12146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 11:15:19.625856 13643 raft_consensus.cc:348] T 8c0e6827f2834d889824e65b185f7a7e P 16c57d96d8364c4e9d5c58b03edb1979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46e12690d5a41f987be0db19d40ccdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39235 } } peers { permanent_uuid: "3e79c692c9cd4d5aae629e060cf997e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "16c57d96d8364c4e9d5c58b03edb1979" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33583 } }
I20260927 11:15:19.626012 13643 raft_consensus.cc:374] T 8c0e6827f2834d889824e65b185f7a7e P 16c57d96d8364c4e9d5c58b03edb1979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:19.626037 13643 raft_consensus.cc:729] T 8c0e6827f2834d889824e65b185f7a7e P 16c57d96d8364c4e9d5c58b03edb1979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16c57d96d8364c4e9d5c58b03edb1979, State: Initialized, Role: FOLLOWER
I20260927 11:15:19.626569 13646 tablet_bootstrap.cc:492] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1: Bootstrap starting.
I20260927 11:15:19.626721 13643 consensus_queue.cc:260] T 8c0e6827f2834d889824e65b185f7a7e P 16c57d96d8364c4e9d5c58b03edb1979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46e12690d5a41f987be0db19d40ccdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39235 } } peers { permanent_uuid: "3e79c692c9cd4d5aae629e060cf997e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "16c57d96d8364c4e9d5c58b03edb1979" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33583 } }
I20260927 11:15:19.627043 13643 ts_tablet_manager.cc:1428] T 8c0e6827f2834d889824e65b185f7a7e P 16c57d96d8364c4e9d5c58b03edb1979: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260927 11:15:19.627180 13341 heartbeater.cc:503] Master 127.0.98.62:39331 was elected leader, sending a full tablet report...
I20260927 11:15:19.627647 13646 tablet_bootstrap.cc:654] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:19.627916 13646 log.cc:824] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:19.628499 13645 tablet_bootstrap.cc:492] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd: Bootstrap starting.
I20260927 11:15:19.629242 13645 tablet_bootstrap.cc:654] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:19.629498 13645 log.cc:824] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:19.630956 13645 tablet_bootstrap.cc:492] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd: No bootstrap required, opened a new log
I20260927 11:15:19.631045 13645 ts_tablet_manager.cc:1397] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 11:15:19.636021 13645 raft_consensus.cc:348] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46e12690d5a41f987be0db19d40ccdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39235 } } peers { permanent_uuid: "3e79c692c9cd4d5aae629e060cf997e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "16c57d96d8364c4e9d5c58b03edb1979" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33583 } }
I20260927 11:15:19.636538 13645 raft_consensus.cc:374] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:19.636607 13645 raft_consensus.cc:729] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46e12690d5a41f987be0db19d40ccdd, State: Initialized, Role: FOLLOWER
I20260927 11:15:19.636787 13645 consensus_queue.cc:260] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46e12690d5a41f987be0db19d40ccdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39235 } } peers { permanent_uuid: "3e79c692c9cd4d5aae629e060cf997e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "16c57d96d8364c4e9d5c58b03edb1979" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33583 } }
I20260927 11:15:19.637243 13609 heartbeater.cc:503] Master 127.0.98.62:39331 was elected leader, sending a full tablet report...
I20260927 11:15:19.637243 13645 ts_tablet_manager.cc:1428] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260927 11:15:19.637974 13646 tablet_bootstrap.cc:492] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1: No bootstrap required, opened a new log
I20260927 11:15:19.638041 13646 ts_tablet_manager.cc:1397] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20260927 11:15:19.640038 13646 raft_consensus.cc:348] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46e12690d5a41f987be0db19d40ccdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39235 } } peers { permanent_uuid: "3e79c692c9cd4d5aae629e060cf997e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "16c57d96d8364c4e9d5c58b03edb1979" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33583 } }
I20260927 11:15:19.640198 13646 raft_consensus.cc:374] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:19.640240 13646 raft_consensus.cc:729] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e79c692c9cd4d5aae629e060cf997e1, State: Initialized, Role: FOLLOWER
I20260927 11:15:19.640357 13646 consensus_queue.cc:260] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46e12690d5a41f987be0db19d40ccdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39235 } } peers { permanent_uuid: "3e79c692c9cd4d5aae629e060cf997e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "16c57d96d8364c4e9d5c58b03edb1979" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33583 } }
I20260927 11:15:19.640610 13646 ts_tablet_manager.cc:1428] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 11:15:19.640635 13475 heartbeater.cc:503] Master 127.0.98.62:39331 was elected leader, sending a full tablet report...
W20260927 11:15:19.665110 13342 tablet.cc:2369] T 8c0e6827f2834d889824e65b185f7a7e P 16c57d96d8364c4e9d5c58b03edb1979: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:15:19.679198 13650 raft_consensus.cc:482] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:15:19.679322 13650 raft_consensus.cc:504] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46e12690d5a41f987be0db19d40ccdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39235 } } peers { permanent_uuid: "3e79c692c9cd4d5aae629e060cf997e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "16c57d96d8364c4e9d5c58b03edb1979" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33583 } }
I20260927 11:15:19.679739 13650 leader_election.cc:288] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e79c692c9cd4d5aae629e060cf997e1 (127.0.98.2:34445), 16c57d96d8364c4e9d5c58b03edb1979 (127.0.98.1:33583)
I20260927 11:15:19.684347 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:19.680146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37189 (local address 127.0.98.1:33583)
0927 11:15:19.680193 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:19.680195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:19.680803 (+ 608us) server_negotiation.cc:407] Connection header received
0927 11:15:19.680901 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:19.680902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:19.680913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:19.680942 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:19.681134 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.681880 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.683318 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.683937 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.683983 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:19.683994 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:19.684225 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 11:15:19.684281 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 11:15:19.684762 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:19.679947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33583 (local address 127.0.98.3:37189)
0927 11:15:19.680758 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:19.680765 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:19.680824 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:19.681002 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:19.681002 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:19.681004 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:19.681067 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:19.681069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.681977 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.681977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:19.683217 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:19.683220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.684075 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.684075 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:19.684146 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:19.684149 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:15:19.684616 (+ 467us) client_negotiation.cc:240] Negotiation successful
0927 11:15:19.684657 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":73,"threads_started":1}
I20260927 11:15:19.685221 13295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0e6827f2834d889824e65b185f7a7e" candidate_uuid: "a46e12690d5a41f987be0db19d40ccdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16c57d96d8364c4e9d5c58b03edb1979" is_pre_election: true
I20260927 11:15:19.685405 13295 raft_consensus.cc:2454] T 8c0e6827f2834d889824e65b185f7a7e P 16c57d96d8364c4e9d5c58b03edb1979 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a46e12690d5a41f987be0db19d40ccdd in term 0.
I20260927 11:15:19.685647 13497 leader_election.cc:302] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16c57d96d8364c4e9d5c58b03edb1979, a46e12690d5a41f987be0db19d40ccdd; no voters:
I20260927 11:15:19.685773 13650 raft_consensus.cc:2781] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:15:19.685810 13650 raft_consensus.cc:482] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:15:19.685832 13650 raft_consensus.cc:3037] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:19.686743 13650 raft_consensus.cc:504] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46e12690d5a41f987be0db19d40ccdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39235 } } peers { permanent_uuid: "3e79c692c9cd4d5aae629e060cf997e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "16c57d96d8364c4e9d5c58b03edb1979" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33583 } }
I20260927 11:15:19.686861 13650 leader_election.cc:288] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [CANDIDATE]: Term 1 election: Requested vote from peers 3e79c692c9cd4d5aae629e060cf997e1 (127.0.98.2:34445), 16c57d96d8364c4e9d5c58b03edb1979 (127.0.98.1:33583)
I20260927 11:15:19.687186 13295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0e6827f2834d889824e65b185f7a7e" candidate_uuid: "a46e12690d5a41f987be0db19d40ccdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16c57d96d8364c4e9d5c58b03edb1979"
I20260927 11:15:19.687252 13295 raft_consensus.cc:3037] T 8c0e6827f2834d889824e65b185f7a7e P 16c57d96d8364c4e9d5c58b03edb1979 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:19.688056 13295 raft_consensus.cc:2454] T 8c0e6827f2834d889824e65b185f7a7e P 16c57d96d8364c4e9d5c58b03edb1979 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a46e12690d5a41f987be0db19d40ccdd in term 1.
I20260927 11:15:19.688258 13497 leader_election.cc:302] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16c57d96d8364c4e9d5c58b03edb1979, a46e12690d5a41f987be0db19d40ccdd; no voters:
I20260927 11:15:19.688360 13650 raft_consensus.cc:2781] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:15:19.688481 13650 raft_consensus.cc:686] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [term 1 LEADER]: Becoming Leader. State: Replica: a46e12690d5a41f987be0db19d40ccdd, State: Running, Role: LEADER
I20260927 11:15:19.688555 13650 consensus_queue.cc:237] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46e12690d5a41f987be0db19d40ccdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39235 } } peers { permanent_uuid: "3e79c692c9cd4d5aae629e060cf997e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } } peers { permanent_uuid: "16c57d96d8364c4e9d5c58b03edb1979" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33583 } }
I20260927 11:15:19.689343 13142 catalog_manager.cc:5495] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd reported cstate change: term changed from 0 to 1, leader changed from <none> to a46e12690d5a41f987be0db19d40ccdd (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a46e12690d5a41f987be0db19d40ccdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46e12690d5a41f987be0db19d40ccdd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3e79c692c9cd4d5aae629e060cf997e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16c57d96d8364c4e9d5c58b03edb1979" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33583 } health_report { overall_health: UNKNOWN } } }
I20260927 11:15:19.700592 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:19.680035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57883 (local address 127.0.98.2:34445)
0927 11:15:19.680094 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:19.680097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:19.680660 (+ 563us) server_negotiation.cc:407] Connection header received
0927 11:15:19.680733 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:19.680734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:19.680746 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:19.680804 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:19.682932 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.683707 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.698177 (+ 14470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.698765 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.698838 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:19.698848 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:19.700468 (+ 1620us) server_negotiation.cc:299] Negotiation successful
0927 11:15:19.700517 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 11:15:19.700989 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:19.679865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34445 (local address 127.0.98.3:57883)
0927 11:15:19.680571 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:19.680583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:19.680659 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:19.680856 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:19.680857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:19.680859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:19.680954 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:19.680956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.684076 (+ 3120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.684077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:19.697229 (+ 13152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:19.697233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.700285 (+ 3052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.700287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:19.700369 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:19.700374 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:15:19.700856 (+ 482us) client_negotiation.cc:240] Negotiation successful
0927 11:15:19.700907 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":77,"threads_started":1}
I20260927 11:15:19.704063 13429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0e6827f2834d889824e65b185f7a7e" candidate_uuid: "a46e12690d5a41f987be0db19d40ccdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e79c692c9cd4d5aae629e060cf997e1"
I20260927 11:15:19.704213 13429 raft_consensus.cc:3037] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:19.705250 13429 raft_consensus.cc:2454] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a46e12690d5a41f987be0db19d40ccdd in term 1.
I20260927 11:15:19.704082 13416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c0e6827f2834d889824e65b185f7a7e" candidate_uuid: "a46e12690d5a41f987be0db19d40ccdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e79c692c9cd4d5aae629e060cf997e1" is_pre_election: true
I20260927 11:15:19.705472 13416 raft_consensus.cc:2362] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a46e12690d5a41f987be0db19d40ccdd in term 1. Re-sending same reply.
I20260927 11:15:19.716514 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:19.708020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52140 (local address 127.0.98.3:39235)
0927 11:15:19.712274 (+ 4254us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:19.712276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:19.712285 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:15:19.712319 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:19.712320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:19.712341 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:19.712416 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:15:19.713117 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.714115 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.715398 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:19.715707 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:19.715764 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:19.715988 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:15:19.716111 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:15:19.716205 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:19.716423 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 11:15:19.716443 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4229}
I20260927 11:15:19.718515 13295 raft_consensus.cc:1261] T 8c0e6827f2834d889824e65b185f7a7e P 16c57d96d8364c4e9d5c58b03edb1979 [term 1 FOLLOWER]: Refusing update from remote peer a46e12690d5a41f987be0db19d40ccdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:15:19.718837 13655 consensus_queue.cc:1035] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "16c57d96d8364c4e9d5c58b03edb1979" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:15:19.720716 13416 raft_consensus.cc:1261] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1 [term 1 FOLLOWER]: Refusing update from remote peer a46e12690d5a41f987be0db19d40ccdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:15:19.720953 13655 consensus_queue.cc:1035] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e79c692c9cd4d5aae629e060cf997e1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:15:19.723315 13658 mvcc.cc:205] Tried to move back new op lower bound from 7333919619964596224 to 7333919619844890624. Current Snapshot: MvccSnapshot[applied={T|T < 7333919619964596224}]
I20260927 11:15:19.726480 13659 mvcc.cc:205] Tried to move back new op lower bound from 7333919619964596224 to 7333919619844890624. Current Snapshot: MvccSnapshot[applied={T|T < 7333919619964596224}]
I20260927 11:15:19.728754 13142 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:15:19.728860 13142 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:15:19.729624 13142 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09e97c5350a04ea19e44b9392ce8f974: Sending DeleteTablet for 3 replicas of tablet 8c0e6827f2834d889824e65b185f7a7e
I20260927 11:15:19.730023 13409 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c0e6827f2834d889824e65b185f7a7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:15:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44046
I20260927 11:15:19.730263 13666 tablet_replica.cc:330] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1: stopping tablet replica
I20260927 11:15:19.730365 13666 raft_consensus.cc:2229] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:15:19.730341 13543 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c0e6827f2834d889824e65b185f7a7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:15:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52132
I20260927 11:15:19.730439 13666 pending_rounds.cc:62] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1: Trying to abort 2 pending ops.
I20260927 11:15:19.730465 13666 pending_rounds.cc:68] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333919619964596224 op_type: WRITE_OP write_request { tablet_id: "8c0e6827f2834d889824e65b185f7a7e" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\363\367\343\325\006\"8\n\ntest-admin\022*\n a47438e214ab44579d6b53fce852a41d\020\001\030\001 \001(\001" signature: "<redacted>""\241\3043\245M\177\300\342\277~\322\323\375\377\234e\256(5\304\211\2463\026\336\327T\334u\314\n5\210\374\023\200\264h\\\210\021#.\205\321(o\025\247,\025\367\352b\256\222\224\261\000_\253\306\333\335" signing_key_seq_num: 0 } } request_id { client_id: "b19ec75bfa0742c89ef4daf25df6827a" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260927 11:15:19.730671 13666 pending_rounds.cc:68] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333919619844890624 op_type: NO_OP noop_request { }
I20260927 11:15:19.730669 13275 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c0e6827f2834d889824e65b185f7a7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:15:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36078
I20260927 11:15:19.730796 13666 raft_consensus.cc:2866] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260927 11:15:19.730850 13666 raft_consensus.cc:2258] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:15:19.731367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 13181
W20260927 11:15:19.733368 13662 consensus_queue.cc:1159] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3e79c692c9cd4d5aae629e060cf997e1" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
W20260927 11:15:19.738291 13496 consensus_peers.cc:487] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd -> Peer 3e79c692c9cd4d5aae629e060cf997e1 (127.0.98.2:34445): Couldn't send request to peer 3e79c692c9cd4d5aae629e060cf997e1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260927 11:15:19.738585 13667 tablet_replica.cc:330] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd: stopping tablet replica
I20260927 11:15:19.738597 13666 ts_tablet_manager.cc:1905] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:15:19.738711 13667 raft_consensus.cc:2229] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:15:19.738862 13667 raft_consensus.cc:2258] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:15:19.739319 13667 ts_tablet_manager.cc:1905] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:15:19.740253 13666 ts_tablet_manager.cc:1918] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:15:19.740417 13666 log.cc:1196] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/8c0e6827f2834d889824e65b185f7a7e
I20260927 11:15:19.740769 13666 ts_tablet_manager.cc:1939] T 8c0e6827f2834d889824e65b185f7a7e P 3e79c692c9cd4d5aae629e060cf997e1: Deleting consensus metadata
I20260927 11:15:19.741277 13126 catalog_manager.cc:4841] TS 3e79c692c9cd4d5aae629e060cf997e1 (127.0.98.2:34445): tablet 8c0e6827f2834d889824e65b185f7a7e (table test-table [id=a47438e214ab44579d6b53fce852a41d]) successfully deleted
I20260927 11:15:19.741321 13667 ts_tablet_manager.cc:1918] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:15:19.741374 13667 log.cc:1196] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/8c0e6827f2834d889824e65b185f7a7e
I20260927 11:15:19.746397 13667 ts_tablet_manager.cc:1939] T 8c0e6827f2834d889824e65b185f7a7e P a46e12690d5a41f987be0db19d40ccdd: Deleting consensus metadata
I20260927 11:15:19.746886 13128 catalog_manager.cc:4841] TS a46e12690d5a41f987be0db19d40ccdd (127.0.98.3:39235): tablet 8c0e6827f2834d889824e65b185f7a7e (table test-table [id=a47438e214ab44579d6b53fce852a41d]) successfully deleted
W20260927 11:15:19.754336 13127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33583 (error 108)
I20260927 11:15:19.758237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 13345
I20260927 11:15:19.760056 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:19.759922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33583 (local address 127.0.0.1:36088)
0927 11:15:19.759984 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:19.760022 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 11:15:19.761051 13127 catalog_manager.cc:4584] TS 16c57d96d8364c4e9d5c58b03edb1979 (127.0.98.1:33583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8c0e6827f2834d889824e65b185f7a7e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33583: connect: Connection refused (error 111)
I20260927 11:15:19.774237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 13478
I20260927 11:15:19.780128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 13109
I20260927 11:15:19.786033 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 11:15:20.119678 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 11:15:20.119762 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 11:15:20.464305 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 11:15:20.464 UTC [12176] LOG: received smart shutdown request
2026-09-27 11:15:20.468 UTC [12176] LOG: background worker "logical replication launcher" (PID 12185) exited with exit code 1
2026-09-27 11:15:20.468 UTC [12180] LOG: shutting down
2026-09-27 11:15:20.611 UTC [12176] LOG: database system is shut down
2026-09-27T11:15:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44543 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:15:20 dist-test-slave-9hq9 krb5kdc[13678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:15:20 dist-test-slave-9hq9 krb5kdc[13678](info): set up 2 sockets
Sep 27 11:15:20 dist-test-slave-9hq9 krb5kdc[13678](info): commencing operation
krb5kdc: starting...
W20260927 11:15:21.745937 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:15:21 dist-test-slave-9hq9 krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:15:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:15:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:15:21.854849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36747
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36747,127.0.98.61:45119,127.0.98.60:37931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:21.932518 13691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:21.932732 13691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:21.932765 13691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:21.933902 13691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:15:21.933965 13691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:21.933988 13691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:21.934010 13691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:15:21.934038 13691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:15:21.935568 13691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36747,127.0.98.61:45119,127.0.98.60:37931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:21.935827 13691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:21.936034 13691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:21.940678 13699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:21.940665 13696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:21.940805 13691 server_base.cc:1034] running on GCE node
W20260927 11:15:21.940665 13697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:21.941198 13691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:21.941483 13691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:21.942638 13691 hybrid_clock.cc:648] HybridClock initialized: now 1790507721942628 us; error 40 us; skew 500 ppm
Sep 27 11:15:21 dist-test-slave-9hq9 krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:21.945032 13691 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:15:21.945441 13691 webserver.cc:460] Webserver started at http://127.0.98.62:43009/ using document root <none> and password file <none>
I20260927 11:15:21.945665 13691 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:21.945730 13691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:21.945873 13691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:15:21.946923 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "f41bb06f73824284ade5f23badc0d59f"
format_stamp: "Formatted at 2026-09-27 11:15:21 on dist-test-slave-9hq9"
server_key: "ba858d7a2a57e22ad825293907861a36"
server_key_iv: "bb667573f2fb9326d1689904ad21bd9e"
server_key_version: "encryptionkey@0"
I20260927 11:15:21.947276 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "f41bb06f73824284ade5f23badc0d59f"
format_stamp: "Formatted at 2026-09-27 11:15:21 on dist-test-slave-9hq9"
server_key: "ba858d7a2a57e22ad825293907861a36"
server_key_iv: "bb667573f2fb9326d1689904ad21bd9e"
server_key_version: "encryptionkey@0"
I20260927 11:15:21.948544 13691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:15:21.949240 13706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:21.949406 13691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:15:21.949493 13691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "f41bb06f73824284ade5f23badc0d59f"
format_stamp: "Formatted at 2026-09-27 11:15:21 on dist-test-slave-9hq9"
server_key: "ba858d7a2a57e22ad825293907861a36"
server_key_iv: "bb667573f2fb9326d1689904ad21bd9e"
server_key_version: "encryptionkey@0"
I20260927 11:15:21.949579 13691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:21.962145 13691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:21.964318 13691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:21.964473 13691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:21.968225 13691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36747
I20260927 11:15:21.968253 13758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36747 every 8 connection(s)
I20260927 11:15:21.968559 13691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:15:21.969228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 13691
I20260927 11:15:21.969331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:15:21.969498 13759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:21.969523 392 external_mini_cluster.cc:1428] Setting key 90afa750007dc800f20f03132dac301c
I20260927 11:15:21.971511 13759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 11:15:21 dist-test-slave-9hq9 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:21.983505 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:21.972184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38558 (local address 127.0.98.62:36747)
0927 11:15:21.972344 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:21.972352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:21.972446 (+ 94us) server_negotiation.cc:407] Connection header received
0927 11:15:21.973197 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:21.973198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:21.973214 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:21.973289 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:21.973916 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:21.975337 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:21.976188 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:21.976688 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:21.976802 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:21.980386 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:21.980394 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:21.980394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:21.980401 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:21.981889 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:21.982375 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:21.982378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:21.982378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:21.982449 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:21.982694 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:21.982695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:21.982695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:21.982952 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:21.983021 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:21.983254 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 11:15:21.983361 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260927 11:15:21.983613 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:21.972133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36747 (local address 127.0.0.1:38558)
0927 11:15:21.972317 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:21.972357 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:21.973038 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:21.973329 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:21.973332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:21.973662 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:21.973817 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:21.973822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:21.975434 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:21.975434 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:21.976088 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:21.976089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:21.976780 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:21.976780 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:21.976830 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:21.977303 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:21.977318 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:21.980309 (+ 2991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:21.981972 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:21.981979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:21.981981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:21.982302 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:21.982517 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:21.982518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:21.982519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:21.982629 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:21.983027 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:21.983035 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:21.983121 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 11:15:21.983230 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 11:15:21.983451 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":71,"thread_start_us":79,"threads_started":1}
I20260927 11:15:21.984319 13759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:15:21.984679 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:21.984549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45119 (local address 127.0.0.1:57518)
0927 11:15:21.984599 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:21.984648 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 11:15:21.984812 13707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45119: connect: Connection refused (error 111)
I20260927 11:15:21.985224 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:21.985138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45119 (local address 127.0.0.1:57520)
0927 11:15:21.985183 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:21.985207 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 11:15:21.985342 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45119: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45119: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 11:15:21.994033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45119
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36747,127.0.98.61:45119,127.0.98.60:37931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:15:22.034595 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } attempt: 1
I20260927 11:15:22.035113 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.034963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45119 (local address 127.0.0.1:57532)
0927 11:15:22.035032 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.035079 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 11:15:22.035487 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.035385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45119 (local address 127.0.0.1:57544)
0927 11:15:22.035427 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.035458 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 11:15:22.035621 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45119: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45119: connect: Connection refused (error 111)
W20260927 11:15:22.071194 13766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:22.071374 13766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:22.071395 13766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:22.072468 13766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:15:22.072506 13766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:22.072520 13766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:22.072531 13766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:15:22.072578 13766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:15:22.074002 13766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36747,127.0.98.61:45119,127.0.98.60:37931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:22.074251 13766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:22.074466 13766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:22.078970 13771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:22.078996 13772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:22.078989 13774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:22.079129 13766 server_base.cc:1034] running on GCE node
I20260927 11:15:22.079371 13766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:22.079618 13766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:22.080772 13766 hybrid_clock.cc:648] HybridClock initialized: now 1790507722080754 us; error 45 us; skew 500 ppm
Sep 27 11:15:22 dist-test-slave-9hq9 krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:22.083024 13766 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 11:15:22.083474 13766 webserver.cc:460] Webserver started at http://127.0.98.61:35919/ using document root <none> and password file <none>
I20260927 11:15:22.083678 13766 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:22.083741 13766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:22.083884 13766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:15:22.084805 13766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/data/instance:
uuid: "7832af67790a4e7bbca546c97909eca0"
format_stamp: "Formatted at 2026-09-27 11:15:22 on dist-test-slave-9hq9"
server_key: "89eba0ea52723a58e913d57cd53e2385"
server_key_iv: "ba727d84520fc7242cd518c45e5a22c1"
server_key_version: "encryptionkey@0"
I20260927 11:15:22.085127 13766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/wal/instance:
uuid: "7832af67790a4e7bbca546c97909eca0"
format_stamp: "Formatted at 2026-09-27 11:15:22 on dist-test-slave-9hq9"
server_key: "89eba0ea52723a58e913d57cd53e2385"
server_key_iv: "ba727d84520fc7242cd518c45e5a22c1"
server_key_version: "encryptionkey@0"
I20260927 11:15:22.086403 13766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:15:22.087041 13781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:22.087200 13766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:15:22.087294 13766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/wal
uuid: "7832af67790a4e7bbca546c97909eca0"
format_stamp: "Formatted at 2026-09-27 11:15:22 on dist-test-slave-9hq9"
server_key: "89eba0ea52723a58e913d57cd53e2385"
server_key_iv: "ba727d84520fc7242cd518c45e5a22c1"
server_key_version: "encryptionkey@0"
I20260927 11:15:22.087366 13766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:22.103787 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } attempt: 2
I20260927 11:15:22.104158 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.104027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45119 (local address 127.0.0.1:57558)
0927 11:15:22.104086 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.104123 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 11:15:22.104470 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.104393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45119 (local address 127.0.0.1:57560)
0927 11:15:22.104431 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.104453 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 11:15:22.104555 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45119: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45119: connect: Connection refused (error 111)
I20260927 11:15:22.110826 13766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:22.112423 13766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:22.112524 13766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:22.116422 13766 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45119
I20260927 11:15:22.116470 13833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45119 every 8 connection(s)
I20260927 11:15:22.116771 13766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/data/info.pb
I20260927 11:15:22.117538 13834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:22.118129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 13766
I20260927 11:15:22.118237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-1/wal/instance
I20260927 11:15:22.118413 392 external_mini_cluster.cc:1428] Setting key a3c18ac078581072c339ff56ff1409af
I20260927 11:15:22.119120 13834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:15:22 dist-test-slave-9hq9 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 11:15:22.128131 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.119788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.0.98.62:36747)
0927 11:15:22.119841 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:22.119844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:22.120042 (+ 198us) server_negotiation.cc:407] Connection header received
0927 11:15:22.120438 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:22.120440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:22.120452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:22.120520 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:22.121071 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.121886 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.122849 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.123234 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.123308 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.125546 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:22.125547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:22.125548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:22.125554 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:22.126986 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.127326 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.127326 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.127327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.127374 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.127613 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.127614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.127614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.127732 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:22.127783 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:22.128008 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 11:15:22.128067 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 11:15:22.128355 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.119726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36747 (local address 127.0.0.1:38568)
0927 11:15:22.119935 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.119961 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:22.120349 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:22.120574 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:22.120576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:22.120796 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:22.120959 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.120963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.121981 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.121981 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.122758 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.122760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.123324 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.123325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.123417 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.123880 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:22.123894 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:22.125463 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:22.127058 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.127064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.127066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.127265 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.127430 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.127431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.127431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.127547 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.127790 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:22.127795 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:22.127898 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 11:15:22.127978 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 11:15:22.128111 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":65,"threads_started":1}
I20260927 11:15:22.128885 13834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:15:22 dist-test-slave-9hq9 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:15:22.135710 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.129097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45119 (local address 127.0.0.1:57574)
0927 11:15:22.129154 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.129164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:22.129200 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:22.129408 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:22.129409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:22.129542 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:22.129576 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.129577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.130501 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.130502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.130973 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.130975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.131438 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.131439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.131500 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.131896 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:22.131901 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:22.132973 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:22.134707 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.134708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.134708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.134864 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.135097 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.135098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.135099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.135150 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.135422 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:22.135423 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:22.135484 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 11:15:22.135554 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 11:15:22.135653 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 11:15:22.135874 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.129168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.0.98.61:45119)
0927 11:15:22.129291 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:22.129295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:22.129308 (+ 13us) server_negotiation.cc:407] Connection header received
0927 11:15:22.129323 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:22.129325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:22.129334 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:22.129409 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:22.129648 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.130396 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.131073 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.131357 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.131439 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.133074 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:22.133081 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:22.133081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:22.133087 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:22.134606 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.134976 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.134979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.134979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.135031 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.135221 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.135222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.135222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.135349 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:22.135432 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:22.135606 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 11:15:22.135713 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260927 11:15:22.136451 13834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:15:22.136735 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.136635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37931 (local address 127.0.0.1:51498)
0927 11:15:22.136680 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.136720 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 11:15:22.136893 13784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37931: connect: Connection refused (error 111)
I20260927 11:15:22.137313 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.137219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37931 (local address 127.0.0.1:51500)
0927 11:15:22.137263 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.137286 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 11:15:22.137401 13834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37931: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37931: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 11:15:22.142330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37931
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36747,127.0.98.61:45119,127.0.98.60:37931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:15:22.186590 13834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } attempt: 1
I20260927 11:15:22.187019 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.186896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37931 (local address 127.0.0.1:51506)
0927 11:15:22.186957 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.186994 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 11:15:22.187405 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.187333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37931 (local address 127.0.0.1:51522)
0927 11:15:22.187368 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.187390 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 11:15:22.187501 13834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37931: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37931: connect: Connection refused (error 111)
I20260927 11:15:22.195678 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } attempt: 3
Sep 27 11:15:22 dist-test-slave-9hq9 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:15:22.202287 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.195991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57586 (local address 127.0.98.61:45119)
0927 11:15:22.196051 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:22.196053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:22.196064 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:15:22.196155 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:22.196156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:22.196171 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:22.196228 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:22.196528 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.197044 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.197716 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.197921 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.198004 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.199925 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:22.199927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:22.199927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:22.199932 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:22.201227 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.201560 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.201562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.201562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.201620 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.201814 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.201815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.201816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.201916 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:22.201988 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:22.202139 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 11:15:22.202206 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 11:15:22.202293 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.195945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45119 (local address 127.0.0.1:57586)
0927 11:15:22.195982 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.195995 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:22.196071 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:22.196233 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:22.196234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:22.196383 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:22.196441 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.196443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.197138 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.197140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.197637 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.197638 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.198005 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.198005 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.198117 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.198556 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:22.198561 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:22.199831 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:22.201319 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.201320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.201320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.201480 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.201692 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.201693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.201693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.201754 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.201980 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:22.201981 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:22.202022 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 11:15:22.202114 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 11:15:22.202205 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 11:15:22.202761 13759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:15:22.203097 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.202977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37931 (local address 127.0.0.1:51534)
0927 11:15:22.203026 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.203050 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 11:15:22.203398 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.203344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37931 (local address 127.0.0.1:51540)
0927 11:15:22.203367 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.203386 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260927 11:15:22.203495 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37931: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37931: connect: Connection refused (error 111)
W20260927 11:15:22.220770 13841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:22.220938 13841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:22.220959 13841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:22.222004 13841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:15:22.222041 13841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:22.222117 13841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:22.222146 13841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:15:22.222158 13841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:15:22.223572 13841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36747,127.0.98.61:45119,127.0.98.60:37931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:22.223738 13841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:22.223889 13841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:15:22.228274 13841 server_base.cc:1034] running on GCE node
W20260927 11:15:22.228258 13847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:22.228224 13849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:22.228248 13846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:22.228724 13841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:22.228986 13841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:22.230110 13841 hybrid_clock.cc:648] HybridClock initialized: now 1790507722230105 us; error 33 us; skew 500 ppm
Sep 27 11:15:22 dist-test-slave-9hq9 krb5kdc[13678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:22.232295 13841 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 11:15:22.232686 13841 webserver.cc:460] Webserver started at http://127.0.98.60:41307/ using document root <none> and password file <none>
I20260927 11:15:22.232929 13841 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:22.232995 13841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:22.233152 13841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:15:22.234192 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/data/instance:
uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8"
format_stamp: "Formatted at 2026-09-27 11:15:22 on dist-test-slave-9hq9"
server_key: "c912145d14c92c5d96007d9cac488dbc"
server_key_iv: "752291e4bf1703a91299c4f0ef2f958c"
server_key_version: "encryptionkey@0"
I20260927 11:15:22.234519 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/wal/instance:
uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8"
format_stamp: "Formatted at 2026-09-27 11:15:22 on dist-test-slave-9hq9"
server_key: "c912145d14c92c5d96007d9cac488dbc"
server_key_iv: "752291e4bf1703a91299c4f0ef2f958c"
server_key_version: "encryptionkey@0"
I20260927 11:15:22.234630 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } attempt: 1
I20260927 11:15:22.234987 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.234850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37931 (local address 127.0.0.1:51546)
0927 11:15:22.234914 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.234949 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 11:15:22.235345 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.235254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37931 (local address 127.0.0.1:51556)
0927 11:15:22.235301 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.235325 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 11:15:22.235474 13759 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37931: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37931: connect: Connection refused (error 111)
I20260927 11:15:22.236125 13841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:15:22.236809 13856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:22.237000 13841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:15:22.237089 13841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/wal
uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8"
format_stamp: "Formatted at 2026-09-27 11:15:22 on dist-test-slave-9hq9"
server_key: "c912145d14c92c5d96007d9cac488dbc"
server_key_iv: "752291e4bf1703a91299c4f0ef2f958c"
server_key_version: "encryptionkey@0"
I20260927 11:15:22.237152 13841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:22.247011 13841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:22.248636 13841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:22.248775 13841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:22.252489 13841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37931
I20260927 11:15:22.252540 13908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37931 every 8 connection(s)
I20260927 11:15:22.252833 13841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/data/info.pb
I20260927 11:15:22.253647 13909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:22.255082 13909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:15:22.255649 13834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } attempt: 2
I20260927 11:15:22.256402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 13841
I20260927 11:15:22.256489 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790507576023819-392-0/minicluster-data/master-2/wal/instance
I20260927 11:15:22.256695 392 external_mini_cluster.cc:1428] Setting key e3383e773ee30677bc2a57b68662a796
Sep 27 11:15:22 dist-test-slave-9hq9 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 11:15:22 dist-test-slave-9hq9 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 11:15:22 dist-test-slave-9hq9 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:15:22.265100 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.255908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37931 (local address 127.0.0.1:51568)
0927 11:15:22.255951 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.255961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:22.256014 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:22.256828 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:22.256829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:22.257028 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:22.257084 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.257086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.258559 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.258560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.259113 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.259115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.259606 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.259607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.259665 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.260087 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:22.260091 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:22.261527 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:22.263741 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.263742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.263742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.263980 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.264257 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.264258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.264259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.264330 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.264778 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:22.264779 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:22.264823 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 11:15:22.264908 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 11:15:22.264996 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 11:15:22.265455 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.255961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.0.98.60:37931)
0927 11:15:22.256095 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:22.256099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:22.256142 (+ 43us) server_negotiation.cc:407] Connection header received
0927 11:15:22.256695 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:22.256705 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:22.256725 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:22.257160 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:22.257246 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.258479 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.259197 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.259527 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.259644 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.261627 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:22.261637 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:22.261637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:22.261644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:22.263661 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.264077 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.264080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.264080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.264176 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.264405 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.264406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.264406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.264692 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:22.264789 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:22.264963 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 11:15:22.265068 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":73,"threads_started":1}
I20260927 11:15:22.265591 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.255691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36747 (local address 127.0.0.1:38578)
0927 11:15:22.255872 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.255907 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:22.256688 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:22.256940 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:22.256943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:22.257323 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:22.257464 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.257469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.258122 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.258123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.259135 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.259137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.259686 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.259687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.259760 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.260307 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:22.260317 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:22.262487 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:22.264241 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.264248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.264250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.264561 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.264775 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.264776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.264776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.264831 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.265125 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:22.265131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:22.265179 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 11:15:22.265287 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 11:15:22.265388 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260927 11:15:22.265843 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.255691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38578 (local address 127.0.98.62:36747)
0927 11:15:22.255734 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:22.255736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:22.256146 (+ 410us) server_negotiation.cc:407] Connection header received
0927 11:15:22.256858 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:22.256859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:22.256870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:22.256920 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:22.257550 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.258003 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.259258 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.259566 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.259628 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.262551 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:22.262553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:22.262553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:22.262559 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:22.264142 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.264645 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.264646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.264647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.264701 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.264905 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.264906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.264907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.265037 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:22.265079 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:22.265662 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 11:15:22.265751 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:15:22.266471 13909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:15:22.266606 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.257028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51584 (local address 127.0.98.60:37931)
0927 11:15:22.257362 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:22.257364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:22.257372 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:15:22.257392 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:22.257392 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:22.257401 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:22.257465 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:22.258200 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.258908 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.259886 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.260201 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.260263 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.262860 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:22.262862 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:22.262862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:22.262868 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:22.264669 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.265043 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.265044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.265044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.265094 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.265490 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.265490 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.265491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.265908 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:22.266083 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:22.266368 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 11:15:22.266445 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":98,"threads_started":1}
I20260927 11:15:22.267499 13834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0: Bootstrap starting.
I20260927 11:15:22.268420 13834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:22.268725 13834 log.cc:824] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:22.269495 13834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0: No bootstrap required, opened a new log
Sep 27 11:15:22 dist-test-slave-9hq9 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:15:22.271562 13834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } }
I20260927 11:15:22.271682 13834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:22.271708 13834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7832af67790a4e7bbca546c97909eca0, State: Initialized, Role: FOLLOWER
I20260927 11:15:22.271811 13834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } }
I20260927 11:15:22.272094 13834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:15:22.272055 13916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } } }
I20260927 11:15:22.272161 13916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:15:22.273504 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.266704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45119 (local address 127.0.0.1:57592)
0927 11:15:22.266765 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.266773 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:22.266803 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:22.266938 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:22.266939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:22.267074 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:22.267125 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.267126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.267711 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.267712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.268474 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.268476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.268867 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.268868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.268938 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.269344 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:22.269348 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:22.270560 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:22.272219 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.272220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.272221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.272386 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.272598 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.272599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.272599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.272682 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.273209 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:22.273209 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:22.273258 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 11:15:22.273333 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 11:15:22.273418 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 11:15:22.273504 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.266756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.0.98.61:45119)
0927 11:15:22.266789 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:22.266791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:22.266798 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:15:22.266860 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:22.266861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:22.266871 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:22.266914 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:22.267193 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.267648 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.268610 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.268807 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.268869 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.270648 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:22.270649 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:22.270650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:22.270656 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:22.272126 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.272468 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.272469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.272469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.272520 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.272756 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.272756 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.272757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.272880 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:22.273212 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:22.273359 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 11:15:22.273425 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:15:22.274184 13909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } has no permanent_uuid. Determining permanent_uuid...
W20260927 11:15:22.274772 13927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:15:22.274873 13927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 11:15:22 dist-test-slave-9hq9 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:22.274230 13766 master_runner.cc:186] Error getting master registration for 127.0.98.62:36747: OK, instance_id { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" instance_seqno: 1790507721967187 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 11:15:22.276410 13766 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 11:15:22.278298 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.269549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38582 (local address 127.0.98.62:36747)
0927 11:15:22.269686 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:22.269688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:22.269694 (+ 6us) server_negotiation.cc:407] Connection header received
0927 11:15:22.269711 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:22.269712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:22.269721 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:22.269777 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:22.270406 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.271069 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.272289 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.272612 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.272685 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.275631 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:22.275632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:22.275632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:22.275640 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:22.277196 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.277507 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.277508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.277508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.277562 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.277762 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.277763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.277763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.277868 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:22.278064 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:22.278120 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 11:15:22.278188 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
Sep 27 11:15:22 dist-test-slave-9hq9 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 11:15:22 dist-test-slave-9hq9 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:15:22.285419 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.276574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37931 (local address 127.0.0.1:51598)
0927 11:15:22.276613 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.276624 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:22.276668 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:22.276791 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:22.276791 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:22.276925 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:22.276955 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.276957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.277771 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.277772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.278535 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.278537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.279001 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.279002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.279044 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.280216 (+ 1172us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:22.280221 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:22.281962 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:22.283749 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.283749 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.283750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.283931 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.284450 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.284451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.284451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.284514 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.285123 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:22.285123 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:22.285179 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 11:15:22.285252 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 11:15:22.285342 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 11:15:22.285528 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.276641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.0.98.60:37931)
0927 11:15:22.276669 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:22.276670 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:22.276678 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:15:22.276724 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:22.276724 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:22.276732 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:22.276786 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:22.277031 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.277688 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.278621 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.278907 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.278962 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.282046 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:22.282082 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:22.282082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:22.282089 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:22.283665 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.284000 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.284001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.284002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.284062 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.284588 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.284589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.284589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.285035 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:22.285112 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:22.285289 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 11:15:22.285402 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":13}
I20260927 11:15:22.286707 13909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8: Bootstrap starting.
I20260927 11:15:22.286748 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.278761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57606 (local address 127.0.98.61:45119)
0927 11:15:22.278794 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:22.278796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:22.278804 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:15:22.278865 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:22.278866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:22.278874 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:22.278924 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:22.279463 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.280107 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.281170 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.281348 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.281420 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.283893 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:22.283894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:22.283894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:22.283901 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:22.285589 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.285936 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.285937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.285937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.285985 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.286229 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.286230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.286230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.286331 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:22.286486 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:22.286533 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 11:15:22.286600 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:15:22.287334 13909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:22.287614 13909 log.cc:824] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:22.287892 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:15:22.288300 13909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8: No bootstrap required, opened a new log
I20260927 11:15:22.290119 13909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } }
I20260927 11:15:22.290252 13909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:22.290287 13909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0de824942d4fa1a5a2d8a22dcfbde8, State: Initialized, Role: FOLLOWER
I20260927 11:15:22.290405 13909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } }
I20260927 11:15:22.290686 13909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 11:15:22.292371 13947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:15:22.292418 13947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:15:22.292853 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } } }
I20260927 11:15:22.292920 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:15:22.292346 13841 master_runner.cc:186] Error getting master registration for 127.0.98.62:36747: OK, instance_id { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" instance_seqno: 1790507721967187 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 11:15:22.294725 13841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 11:15:22.297366 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.289149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.0.98.60:37931)
0927 11:15:22.289182 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:22.289184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:22.289532 (+ 348us) server_negotiation.cc:407] Connection header received
0927 11:15:22.289622 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:22.289623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:22.289633 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:22.289661 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:22.290373 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.291107 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.292230 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.292496 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.292546 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.294337 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:22.294339 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:22.294339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:22.294346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:22.296075 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.296462 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.296463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.296463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.296515 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.296728 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.296729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.296729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.296849 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:22.296935 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:22.297186 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 11:15:22.297256 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:15:22.298969 13838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.289269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57614 (local address 127.0.98.61:45119)
0927 11:15:22.289302 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:22.289305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:22.289376 (+ 71us) server_negotiation.cc:407] Connection header received
0927 11:15:22.289441 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:22.289442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:22.289451 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:22.289502 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:22.290199 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.291120 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.292176 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.292536 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.292573 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.294387 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:22.294388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:22.294388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:22.294393 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:22.296861 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.297338 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.297339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.297340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.297409 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.298412 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.298413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.298413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.298539 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:22.298604 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:22.298803 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 11:15:22.298881 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:15:22.300191 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.291466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.0.98.62:36747)
0927 11:15:22.291496 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:22.291498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:22.293105 (+ 1607us) server_negotiation.cc:407] Connection header received
0927 11:15:22.293132 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:22.293133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:22.293144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:22.293204 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:22.293880 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.294561 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.295576 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.295864 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.295902 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.297310 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:22.297311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:22.297311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:22.297317 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:22.298933 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.299340 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.299341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.299341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.299390 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.299580 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.299581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.299581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.299699 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:22.299776 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:22.300004 (+ 228us) server_negotiation.cc:299] Negotiation successful
0927 11:15:22.300092 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 11:15:22.310638 13759 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } attempt: 2
Sep 27 11:15:22 dist-test-slave-9hq9 krb5kdc[13678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:15:22.318382 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.310907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37931 (local address 127.0.0.1:51614)
0927 11:15:22.310981 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:22.310992 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:22.311043 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:22.311189 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:22.311190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:22.311346 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:22.311386 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.311387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.312264 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.312265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.313083 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:22.313085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.313623 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.313624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:22.313684 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.314123 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:22.314128 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:22.315557 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:22.317314 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.317315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.317315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.317524 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.317733 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:22.317734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:22.317734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:22.317796 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:22.318088 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:22.318089 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:22.318132 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 11:15:22.318204 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 11:15:22.318286 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 11:15:22.318395 13914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:22.310972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51614 (local address 127.0.98.60:37931)
0927 11:15:22.311009 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:22.311011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:22.311022 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:15:22.311109 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:22.311109 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:22.311118 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:22.311176 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:22.311455 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.312185 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.313253 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:22.313547 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:22.313612 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:22.315642 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:22.315643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:22.315644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:22.315650 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:22.317229 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.317607 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.317608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.317609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.317664 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:22.317861 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:22.317861 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:22.317862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:22.317980 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:22.318074 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:22.318239 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 11:15:22.318309 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:15:22.320142 13759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f: Bootstrap starting.
I20260927 11:15:22.320818 13759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:22.321102 13759 log.cc:824] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:22.321919 13759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f: No bootstrap required, opened a new log
I20260927 11:15:22.323923 13759 raft_consensus.cc:348] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } }
I20260927 11:15:22.324115 13759 raft_consensus.cc:374] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:22.324193 13759 raft_consensus.cc:729] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41bb06f73824284ade5f23badc0d59f, State: Initialized, Role: FOLLOWER
I20260927 11:15:22.324368 13759 consensus_queue.cc:260] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } }
I20260927 11:15:22.324713 13759 sys_catalog.cc:564] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f [sys.catalog]: configured and running, proceeding with master startup.
W20260927 11:15:22.326741 13962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:15:22.326789 13962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:15:22.329038 13691 master_runner.cc:263] No leader master found from master f41bb06f73824284ade5f23badc0d59f
I20260927 11:15:22.329083 13691 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 11:15:22.330164 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } } }
I20260927 11:15:22.330299 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f [sys.catalog]: This master's current role is: FOLLOWER
W20260927 11:15:22.363085 13787 tablet.cc:2369] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:15:22.465226 13712 tablet.cc:2369] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:15:22.499336 13862 tablet.cc:2369] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:15:22.534614 13916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:15:22.534722 13916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } }
I20260927 11:15:22.534925 13916 leader_election.cc:288] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f41bb06f73824284ade5f23badc0d59f (127.0.98.62:36747), 8e0de824942d4fa1a5a2d8a22dcfbde8 (127.0.98.60:37931)
I20260927 11:15:22.535168 13883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7832af67790a4e7bbca546c97909eca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" is_pre_election: true
I20260927 11:15:22.535197 13733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7832af67790a4e7bbca546c97909eca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41bb06f73824284ade5f23badc0d59f" is_pre_election: true
I20260927 11:15:22.535323 13733 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7832af67790a4e7bbca546c97909eca0 in term 0.
I20260927 11:15:22.535327 13883 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7832af67790a4e7bbca546c97909eca0 in term 0.
I20260927 11:15:22.535483 13783 leader_election.cc:302] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7832af67790a4e7bbca546c97909eca0, f41bb06f73824284ade5f23badc0d59f; no voters:
I20260927 11:15:22.535634 13916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:15:22.535701 13916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:15:22.535737 13916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:22.536478 13916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } }
I20260927 11:15:22.536636 13916 leader_election.cc:288] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [CANDIDATE]: Term 1 election: Requested vote from peers f41bb06f73824284ade5f23badc0d59f (127.0.98.62:36747), 8e0de824942d4fa1a5a2d8a22dcfbde8 (127.0.98.60:37931)
I20260927 11:15:22.536744 13733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7832af67790a4e7bbca546c97909eca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41bb06f73824284ade5f23badc0d59f"
I20260927 11:15:22.536836 13733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:22.536823 13883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7832af67790a4e7bbca546c97909eca0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8"
I20260927 11:15:22.536911 13883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:22.537607 13883 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7832af67790a4e7bbca546c97909eca0 in term 1.
I20260927 11:15:22.537686 13733 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7832af67790a4e7bbca546c97909eca0 in term 1.
I20260927 11:15:22.537765 13784 leader_election.cc:302] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7832af67790a4e7bbca546c97909eca0, 8e0de824942d4fa1a5a2d8a22dcfbde8; no voters:
I20260927 11:15:22.537889 13916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:15:22.538086 13916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [term 1 LEADER]: Becoming Leader. State: Replica: 7832af67790a4e7bbca546c97909eca0, State: Running, Role: LEADER
I20260927 11:15:22.538177 13916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } }
I20260927 11:15:22.538595 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7832af67790a4e7bbca546c97909eca0. Latest consensus state: current_term: 1 leader_uuid: "7832af67790a4e7bbca546c97909eca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } } }
I20260927 11:15:22.538689 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:22.538909 13966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:15:22.539086 13966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:15:22.540457 13733 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f [term 1 FOLLOWER]: Refusing update from remote peer 7832af67790a4e7bbca546c97909eca0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:15:22.540534 13883 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8 [term 1 FOLLOWER]: Refusing update from remote peer 7832af67790a4e7bbca546c97909eca0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:15:22.540738 13964 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:15:22.540875 13916 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:15:22.541971 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7832af67790a4e7bbca546c97909eca0. Latest consensus state: current_term: 1 leader_uuid: "7832af67790a4e7bbca546c97909eca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } } }
I20260927 11:15:22.541996 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7832af67790a4e7bbca546c97909eca0. Latest consensus state: current_term: 1 leader_uuid: "7832af67790a4e7bbca546c97909eca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } } }
I20260927 11:15:22.542110 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:15:22.542124 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:15:22.542773 13967 mvcc.cc:205] Tried to move back new op lower bound from 7333919631523577856 to 7333919631517003776. Current Snapshot: MvccSnapshot[applied={T|T < 7333919631523577856}]
I20260927 11:15:22.542934 13916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7832af67790a4e7bbca546c97909eca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } } }
I20260927 11:15:22.542989 13969 mvcc.cc:205] Tried to move back new op lower bound from 7333919631523577856 to 7333919631517003776. Current Snapshot: MvccSnapshot[applied={T|T < 7333919631523577856}]
I20260927 11:15:22.543013 13916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:22.543051 13951 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7832af67790a4e7bbca546c97909eca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } } }
I20260927 11:15:22.543164 13951 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:15:22.543313 13964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7832af67790a4e7bbca546c97909eca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } } }
I20260927 11:15:22.543443 13936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7832af67790a4e7bbca546c97909eca0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41bb06f73824284ade5f23badc0d59f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36747 } } peers { permanent_uuid: "7832af67790a4e7bbca546c97909eca0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45119 } } peers { permanent_uuid: "8e0de824942d4fa1a5a2d8a22dcfbde8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37931 } } }
I20260927 11:15:22.543401 13964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0 [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:22.543522 13936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:15:22.543495 13966 catalog_manager.cc:1347] Generated new cluster ID: 0e88ca5aaf5e42b2932117691adc587c
I20260927 11:15:22.543558 13966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:15:22.543677 13968 mvcc.cc:205] Tried to move back new op lower bound from 7333919631523577856 to 7333919631517003776. Current Snapshot: MvccSnapshot[applied={T|T < 7333919631523577856}]
I20260927 11:15:22.548854 13966 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:15:22.549405 13966 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:15:22.558132 13966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7832af67790a4e7bbca546c97909eca0: Generated new TSK 0
I20260927 11:15:22.558319 13966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:15:23.292815 13947 catalog_manager.cc:1259] Loaded cluster ID: 0e88ca5aaf5e42b2932117691adc587c
I20260927 11:15:23.292904 13947 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8: loading cluster ID for follower catalog manager: success
I20260927 11:15:23.293704 13947 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8: acquiring CA information for follower catalog manager: success
I20260927 11:15:23.294011 13947 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e0de824942d4fa1a5a2d8a22dcfbde8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 11:15:23.327184 13962 catalog_manager.cc:1259] Loaded cluster ID: 0e88ca5aaf5e42b2932117691adc587c
I20260927 11:15:23.327275 13962 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f: loading cluster ID for follower catalog manager: success
I20260927 11:15:23.328047 13962 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f: acquiring CA information for follower catalog manager: success
I20260927 11:15:23.328342 13962 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f41bb06f73824284ade5f23badc0d59f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 11:15:23.619820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 13691
I20260927 11:15:23.624722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 13766
I20260927 11:15:23.629140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 13841
2026-09-27T11:15:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2911 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:15:23 dist-test-slave-9hq9 krb5kdc[13982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:15:23 dist-test-slave-9hq9 krb5kdc[13982](info): set up 2 sockets
Sep 27 11:15:23 dist-test-slave-9hq9 krb5kdc[13982](info): commencing operation
krb5kdc: starting...
W20260927 11:15:24.666196 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:15:24 dist-test-slave-9hq9 krb5kdc[13982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:15:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:15:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:15:24.774971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45997
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:24.851899 13995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:24.852075 13995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:24.852097 13995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:24.853209 13995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:15:24.853247 13995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:24.853261 13995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:24.853274 13995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:15:24.853327 13995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:15:24.854773 13995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:24.854997 13995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:24.855211 13995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:24.859675 14000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:24.859740 14001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:24.859687 14003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:24.859948 13995 server_base.cc:1034] running on GCE node
I20260927 11:15:24.860138 13995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:24.860405 13995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:24.861549 13995 hybrid_clock.cc:648] HybridClock initialized: now 1790507724861536 us; error 29 us; skew 500 ppm
Sep 27 11:15:24 dist-test-slave-9hq9 krb5kdc[13982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:24.863842 13995 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:15:24.864239 13995 webserver.cc:460] Webserver started at http://127.0.98.62:34513/ using document root <none> and password file <none>
I20260927 11:15:24.864449 13995 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:24.864490 13995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:24.864631 13995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:15:24.865609 13995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "9b93e53480024d0aada9b122b36dd3cc"
format_stamp: "Formatted at 2026-09-27 11:15:24 on dist-test-slave-9hq9"
server_key: "626ef4ca5cf29e1687c92fa85b26ed02"
server_key_iv: "430e0502263222d5577b99dc2aed529c"
server_key_version: "encryptionkey@0"
I20260927 11:15:24.865947 13995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b93e53480024d0aada9b122b36dd3cc"
format_stamp: "Formatted at 2026-09-27 11:15:24 on dist-test-slave-9hq9"
server_key: "626ef4ca5cf29e1687c92fa85b26ed02"
server_key_iv: "430e0502263222d5577b99dc2aed529c"
server_key_version: "encryptionkey@0"
I20260927 11:15:24.867168 13995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:15:24.867811 14010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:24.867964 13995 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:15:24.868052 13995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "9b93e53480024d0aada9b122b36dd3cc"
format_stamp: "Formatted at 2026-09-27 11:15:24 on dist-test-slave-9hq9"
server_key: "626ef4ca5cf29e1687c92fa85b26ed02"
server_key_iv: "430e0502263222d5577b99dc2aed529c"
server_key_version: "encryptionkey@0"
I20260927 11:15:24.868127 13995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:24.885643 13995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:24.887398 13995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:24.887559 13995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:24.891413 13995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45997
I20260927 11:15:24.891448 14062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45997 every 8 connection(s)
I20260927 11:15:24.891734 13995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:15:24.892455 14063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:24.894502 14063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc: Bootstrap starting.
I20260927 11:15:24.895081 14063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:24.895357 14063 log.cc:824] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:24.895984 14063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc: No bootstrap required, opened a new log
I20260927 11:15:24.897468 14063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b93e53480024d0aada9b122b36dd3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45997 } }
I20260927 11:15:24.897589 14063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:24.897611 14063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b93e53480024d0aada9b122b36dd3cc, State: Initialized, Role: FOLLOWER
I20260927 11:15:24.897789 14063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b93e53480024d0aada9b122b36dd3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45997 } }
I20260927 11:15:24.897871 14063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:15:24.897893 14063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:15:24.897959 14063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:24.898474 14063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b93e53480024d0aada9b122b36dd3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45997 } }
I20260927 11:15:24.898567 14063 leader_election.cc:302] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b93e53480024d0aada9b122b36dd3cc; no voters:
I20260927 11:15:24.898702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 13995
I20260927 11:15:24.898759 14063 leader_election.cc:288] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:15:24.898778 14066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:15:24.898815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:15:24.898891 14066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [term 1 LEADER]: Becoming Leader. State: Replica: 9b93e53480024d0aada9b122b36dd3cc, State: Running, Role: LEADER
I20260927 11:15:24.898958 14066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b93e53480024d0aada9b122b36dd3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45997 } }
I20260927 11:15:24.899003 392 external_mini_cluster.cc:1428] Setting key 4844dee076d8b43cade30582710cc728
I20260927 11:15:24.899008 14063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:15:24.899245 14067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b93e53480024d0aada9b122b36dd3cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b93e53480024d0aada9b122b36dd3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45997 } } }
I20260927 11:15:24.899322 14067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:24.899471 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b93e53480024d0aada9b122b36dd3cc. Latest consensus state: current_term: 1 leader_uuid: "9b93e53480024d0aada9b122b36dd3cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b93e53480024d0aada9b122b36dd3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45997 } } }
I20260927 11:15:24.899533 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:24.899639 14071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:15:24.900125 14071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:15:24.902930 14071 catalog_manager.cc:1347] Generated new cluster ID: 2907147a2ca44d0d908444c1fc348bcb
I20260927 11:15:24.902981 14071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:15:24 dist-test-slave-9hq9 krb5kdc[13982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:24.910353 14072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:24.899397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.0.98.62:45997)
0927 11:15:24.899830 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:24.899838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:24.899872 (+ 34us) server_negotiation.cc:407] Connection header received
0927 11:15:24.899982 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:24.899988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:24.900043 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:24.901323 (+ 1280us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:24.901401 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:24.902309 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:24.903362 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:24.903942 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:24.904042 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:24.906736 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:24.906740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:24.906741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:24.906751 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:24.908729 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:24.909111 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:24.909113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:24.909114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:24.909242 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:24.909445 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:24.909446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:24.909446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:24.909749 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:24.909900 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:24.909978 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 11:15:24.910135 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":79,"threads_started":1}
I20260927 11:15:24.911226 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:15:24.921480 14071 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:15:24.922149 14071 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:15:24.929378 14071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b93e53480024d0aada9b122b36dd3cc: Generated new TSK 0
I20260927 11:15:24.929533 14071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:15:24.964902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 13995
2026-09-27T11:15:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1334 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:15:25 dist-test-slave-9hq9 krb5kdc[14090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:15:25 dist-test-slave-9hq9 krb5kdc[14090](info): set up 2 sockets
Sep 27 11:15:25 dist-test-slave-9hq9 krb5kdc[14090](info): commencing operation
krb5kdc: starting...
W20260927 11:15:26.002079 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:15:26 dist-test-slave-9hq9 krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:15:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:15:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:15:26.109005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38695
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:26.185501 14103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:26.185734 14103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:26.185771 14103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:26.186874 14103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:15:26.186936 14103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:26.186957 14103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:26.186977 14103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:15:26.187006 14103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:15:26.188392 14103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:26.188632 14103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:26.188860 14103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:26.193426 14111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:26.193462 14108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:26.193570 14103 server_base.cc:1034] running on GCE node
W20260927 11:15:26.193459 14109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:26.193850 14103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:26.194181 14103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:26.195335 14103 hybrid_clock.cc:648] HybridClock initialized: now 1790507726195330 us; error 35 us; skew 500 ppm
Sep 27 11:15:26 dist-test-slave-9hq9 krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:26.200982 14103 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:15:26.201414 14103 webserver.cc:460] Webserver started at http://127.0.98.62:44969/ using document root <none> and password file <none>
I20260927 11:15:26.201638 14103 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:26.201705 14103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:26.201859 14103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:15:26.202869 14103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "37a2d40778c94662958cac084069641e"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "8bed7fe6207f7d154899ba11e56bc9ed"
server_key_iv: "6307da24148db027e3ce6b1d5d0aa3de"
server_key_version: "encryptionkey@0"
I20260927 11:15:26.203207 14103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "37a2d40778c94662958cac084069641e"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "8bed7fe6207f7d154899ba11e56bc9ed"
server_key_iv: "6307da24148db027e3ce6b1d5d0aa3de"
server_key_version: "encryptionkey@0"
I20260927 11:15:26.204434 14103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:15:26.205119 14118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:26.205291 14103 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:15:26.205387 14103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "37a2d40778c94662958cac084069641e"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "8bed7fe6207f7d154899ba11e56bc9ed"
server_key_iv: "6307da24148db027e3ce6b1d5d0aa3de"
server_key_version: "encryptionkey@0"
I20260927 11:15:26.205470 14103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:26.215293 14103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:26.217427 14103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:26.217585 14103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:26.221550 14103 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38695
I20260927 11:15:26.221582 14170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38695 every 8 connection(s)
I20260927 11:15:26.221895 14103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:15:26.222828 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 14103
I20260927 11:15:26.222914 14171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:26.222930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:15:26.223124 392 external_mini_cluster.cc:1428] Setting key a1c755cc0a55573f62b3903bcf41e3c7
I20260927 11:15:26.226027 14171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: Bootstrap starting.
I20260927 11:15:26.226660 14171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:26.226931 14171 log.cc:824] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:26.227658 14171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: No bootstrap required, opened a new log
I20260927 11:15:26.229579 14171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } }
I20260927 11:15:26.229743 14171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:26.229775 14171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a2d40778c94662958cac084069641e, State: Initialized, Role: FOLLOWER
Sep 27 11:15:26 dist-test-slave-9hq9 krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:26.229883 14171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } }
I20260927 11:15:26.229955 14171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:15:26.229997 14171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:15:26.230078 14171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:26.230721 14171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } }
I20260927 11:15:26.230831 14171 leader_election.cc:302] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37a2d40778c94662958cac084069641e; no voters:
I20260927 11:15:26.231004 14171 leader_election.cc:288] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:15:26.231034 14176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:15:26.231261 14171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:15:26.231392 14176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 1 LEADER]: Becoming Leader. State: Replica: 37a2d40778c94662958cac084069641e, State: Running, Role: LEADER
I20260927 11:15:26.231525 14176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } }
I20260927 11:15:26.231853 14177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37a2d40778c94662958cac084069641e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } } }
I20260927 11:15:26.231957 14177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:26.232121 14178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37a2d40778c94662958cac084069641e. Latest consensus state: current_term: 1 leader_uuid: "37a2d40778c94662958cac084069641e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } } }
I20260927 11:15:26.232152 14185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:15:26.232177 14178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:26.232697 14185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:15:26.234038 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.223733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.0.98.62:38695)
0927 11:15:26.223896 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:26.223905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:26.223935 (+ 30us) server_negotiation.cc:407] Connection header received
0927 11:15:26.224421 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:26.224430 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:26.224490 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:26.224651 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:26.225208 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.226531 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.227445 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.228017 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.228126 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.230436 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:26.230439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:26.230439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:26.230447 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:26.232366 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:26.232791 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:26.232793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:26.232794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:26.232885 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:26.233100 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:26.233101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:26.233101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:26.233393 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:26.233554 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:26.233722 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 11:15:26.233858 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260927 11:15:26.238464 14185 catalog_manager.cc:1347] Generated new cluster ID: 8551c1fc6c8d4fbc961853a03fdde03e
I20260927 11:15:26.238538 14185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:15:26.242815 14185 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:15:26.243314 14185 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:15:26.248756 14185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: Generated new TSK 0
I20260927 11:15:26.248916 14185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:15:26.273914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38695
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:26.350462 14197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:26.350638 14197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:26.350660 14197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:26.351773 14197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:26.351811 14197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:26.351836 14197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:15:26.353214 14197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:26.353435 14197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:26.353655 14197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:26.358352 14202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:26.358484 14197 server_base.cc:1034] running on GCE node
W20260927 11:15:26.358327 14205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:26.358338 14203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:26.358816 14197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:26.359020 14197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:26.360155 14197 hybrid_clock.cc:648] HybridClock initialized: now 1790507726360144 us; error 32 us; skew 500 ppm
Sep 27 11:15:26 dist-test-slave-9hq9 krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:26.362438 14197 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:15:26.362840 14197 webserver.cc:460] Webserver started at http://127.0.98.1:46221/ using document root <none> and password file <none>
I20260927 11:15:26.363066 14197 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:26.363145 14197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:26.363293 14197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:15:26.364236 14197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "220afa49dff44ded9274015f228645f8"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "90e5e6b06cf5b124c32b4b3807f9a8e3"
server_key_iv: "dbe0b28883742441a6e4b311198f9609"
server_key_version: "encryptionkey@0"
I20260927 11:15:26.364575 14197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "220afa49dff44ded9274015f228645f8"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "90e5e6b06cf5b124c32b4b3807f9a8e3"
server_key_iv: "dbe0b28883742441a6e4b311198f9609"
server_key_version: "encryptionkey@0"
I20260927 11:15:26.365774 14197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:15:26.366487 14212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:26.366652 14197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:26.366750 14197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "220afa49dff44ded9274015f228645f8"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "90e5e6b06cf5b124c32b4b3807f9a8e3"
server_key_iv: "dbe0b28883742441a6e4b311198f9609"
server_key_version: "encryptionkey@0"
I20260927 11:15:26.366832 14197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:26.382400 14197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:26.384079 14197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:26.384227 14197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:26.384436 14197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:26.384721 14197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:15:26.384773 14197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:26.384812 14197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:15:26.384840 14197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:26.390535 14197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43815
I20260927 11:15:26.390571 14325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43815 every 8 connection(s)
I20260927 11:15:26.390878 14197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:15:26 dist-test-slave-9hq9 krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:26.397939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 14197
I20260927 11:15:26.398030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:15:26.398255 392 external_mini_cluster.cc:1428] Setting key bacfcc9a46df9b0ee90161122dd382c9
I20260927 11:15:26.399925 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.391573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:56253)
0927 11:15:26.391728 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:26.391759 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:26.392219 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:26.392467 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:26.392470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:26.392988 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:26.393159 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.393163 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.393860 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.393861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.394701 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.394703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.395104 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.395105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.395215 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.395677 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:26.395684 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:26.396952 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:26.398603 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:26.398610 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:26.398612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:26.398919 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:26.399131 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:26.399132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:26.399132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:26.399231 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:26.399499 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:26.399502 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:26.399601 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 11:15:26.399678 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 11:15:26.399785 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260927 11:15:26.399946 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.391584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56253 (local address 127.0.98.62:38695)
0927 11:15:26.391642 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:26.391644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:26.391843 (+ 199us) server_negotiation.cc:407] Connection header received
0927 11:15:26.392318 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:26.392319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:26.392331 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:26.392394 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:26.393262 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.393766 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.394795 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.395014 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.395086 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.397044 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:26.397046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:26.397046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:26.397054 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:26.398459 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:26.399002 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:26.399003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:26.399003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:26.399063 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:26.399291 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:26.399292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:26.399292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:26.399435 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:26.399505 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:26.399718 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 11:15:26.399808 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 11:15:26.400364 14326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38695
I20260927 11:15:26.400455 14326 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:26.400616 14326 heartbeater.cc:511] Master 127.0.98.62:38695 requested a full tablet report, sending...
I20260927 11:15:26.401126 14135 ts_manager.cc:194] Registered new tserver with Master: 220afa49dff44ded9274015f228645f8 (127.0.98.1:43815)
I20260927 11:15:26.402036 14135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56253
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:15:26.424582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38695
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:26.502985 14331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:26.503185 14331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:26.503208 14331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:26.504442 14331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:26.504522 14331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:26.504613 14331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:15:26.506090 14331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:26.506323 14331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:26.506547 14331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:26.511521 14337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:26.511646 14331 server_base.cc:1034] running on GCE node
W20260927 11:15:26.511548 14336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:26.511513 14339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:26.511952 14331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:26.512147 14331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:26.513298 14331 hybrid_clock.cc:648] HybridClock initialized: now 1790507726513289 us; error 28 us; skew 500 ppm
Sep 27 11:15:26 dist-test-slave-9hq9 krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:26.515784 14331 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:15:26.516172 14331 webserver.cc:460] Webserver started at http://127.0.98.2:34873/ using document root <none> and password file <none>
I20260927 11:15:26.516377 14331 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:26.516418 14331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:26.516552 14331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:15:26.517462 14331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "e04011c612004fabbdd89cefc40e4083"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "2a19acea7d8b405a1890b79b94084975"
server_key_iv: "4118f28a9f6b3fa575952cd5704bfd69"
server_key_version: "encryptionkey@0"
I20260927 11:15:26.517787 14331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e04011c612004fabbdd89cefc40e4083"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "2a19acea7d8b405a1890b79b94084975"
server_key_iv: "4118f28a9f6b3fa575952cd5704bfd69"
server_key_version: "encryptionkey@0"
I20260927 11:15:26.519105 14331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:15:26.519768 14346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:26.519915 14331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:15:26.520001 14331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "e04011c612004fabbdd89cefc40e4083"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "2a19acea7d8b405a1890b79b94084975"
server_key_iv: "4118f28a9f6b3fa575952cd5704bfd69"
server_key_version: "encryptionkey@0"
I20260927 11:15:26.520081 14331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:26.549724 14331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:26.551860 14331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:26.552007 14331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:26.552222 14331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:26.552488 14331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:15:26.552538 14331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:26.552577 14331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:15:26.552603 14331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:26.557936 14331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40749
I20260927 11:15:26.557974 14459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40749 every 8 connection(s)
I20260927 11:15:26.558290 14331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:15:26.559454 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 14331
I20260927 11:15:26.559588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:15:26.559758 392 external_mini_cluster.cc:1428] Setting key 003386c057a16a7032ba9db1be22635f
Sep 27 11:15:26 dist-test-slave-9hq9 krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:26.566941 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.558923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59549 (local address 127.0.98.62:38695)
0927 11:15:26.558978 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:26.558980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:26.559149 (+ 169us) server_negotiation.cc:407] Connection header received
0927 11:15:26.559644 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:26.559646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:26.559657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:26.559720 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:26.560739 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.561210 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.562153 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.562352 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.562426 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.564413 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:26.564415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:26.564415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:26.564423 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:26.565707 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:26.566074 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:26.566076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:26.566076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:26.566133 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:26.566361 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:26.566362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:26.566362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:26.566469 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:26.566538 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:26.566781 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 11:15:26.566855 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 11:15:26.567108 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.558909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:59549)
0927 11:15:26.559048 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:26.559078 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:26.559544 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:26.559787 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:26.559790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:26.560477 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:26.560649 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.560654 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.561295 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.561296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.562042 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.562044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.562435 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.562436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.562564 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.563001 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:26.563009 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:26.564329 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:26.565782 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:26.565789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:26.565791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:26.565991 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:26.566200 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:26.566201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:26.566201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:26.566300 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:26.566530 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:26.566536 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:26.566662 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 11:15:26.566761 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 11:15:26.566902 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 11:15:26.567651 14460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38695
I20260927 11:15:26.567744 14460 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:26.567896 14460 heartbeater.cc:511] Master 127.0.98.62:38695 requested a full tablet report, sending...
I20260927 11:15:26.568266 14135 ts_manager.cc:194] Registered new tserver with Master: e04011c612004fabbdd89cefc40e4083 (127.0.98.2:40749)
I20260927 11:15:26.568619 14135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:15:26.582554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38695
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:26.661175 14465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:26.661369 14465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:26.661391 14465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:26.662467 14465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:26.662500 14465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:26.662532 14465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:15:26.663859 14465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:26.664026 14465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:26.664193 14465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:26.669068 14471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:26.669049 14470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:26.669113 14465 server_base.cc:1034] running on GCE node
W20260927 11:15:26.669034 14473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:26.669549 14465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:26.669792 14465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:26.670941 14465 hybrid_clock.cc:648] HybridClock initialized: now 1790507726670931 us; error 34 us; skew 500 ppm
Sep 27 11:15:26 dist-test-slave-9hq9 krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:26.673151 14465 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:15:26.673532 14465 webserver.cc:460] Webserver started at http://127.0.98.3:46549/ using document root <none> and password file <none>
I20260927 11:15:26.673760 14465 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:26.673825 14465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:26.673969 14465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:15:26.674993 14465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "b31be1cacc454e18a5cc67d33cab6c61"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "3588a434c9b3fad7e5e8835ada0e327b"
server_key_iv: "d0b3b4711cc7ad0b94360f2724dfdd6d"
server_key_version: "encryptionkey@0"
I20260927 11:15:26.675333 14465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b31be1cacc454e18a5cc67d33cab6c61"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "3588a434c9b3fad7e5e8835ada0e327b"
server_key_iv: "d0b3b4711cc7ad0b94360f2724dfdd6d"
server_key_version: "encryptionkey@0"
I20260927 11:15:26.676574 14465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:15:26.677204 14480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:26.677354 14465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:15:26.677445 14465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "b31be1cacc454e18a5cc67d33cab6c61"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "3588a434c9b3fad7e5e8835ada0e327b"
server_key_iv: "d0b3b4711cc7ad0b94360f2724dfdd6d"
server_key_version: "encryptionkey@0"
I20260927 11:15:26.677520 14465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:26.685793 14465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:26.687443 14465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:26.687582 14465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:26.687777 14465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:26.688051 14465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:15:26.688100 14465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:26.688133 14465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:15:26.688158 14465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:26.693558 14465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45691
I20260927 11:15:26.693590 14593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45691 every 8 connection(s)
I20260927 11:15:26.693863 14465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:15:26.696575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 14465
I20260927 11:15:26.696668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:15:26.696831 392 external_mini_cluster.cc:1428] Setting key 1fa28e1ee399d0fdcfc2a970f0241851
Sep 27 11:15:26 dist-test-slave-9hq9 krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:26.702742 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.694548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38891 (local address 127.0.98.62:38695)
0927 11:15:26.694592 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:26.694594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:26.694794 (+ 200us) server_negotiation.cc:407] Connection header received
0927 11:15:26.695260 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:26.695261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:26.695272 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:26.695329 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:26.696188 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.696882 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.698022 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.698259 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.698310 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.700086 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:26.700087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:26.700088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:26.700095 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:26.701418 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:26.701808 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:26.701809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:26.701810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:26.701864 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:26.702124 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:26.702125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:26.702125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:26.702232 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:26.702303 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:26.702571 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 11:15:26.702657 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:15:26.702880 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.694512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:38891)
0927 11:15:26.694671 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:26.694726 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:26.695163 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:26.695397 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:26.695401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:26.695916 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:26.696080 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.696083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.696963 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.696964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.697960 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.697963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.698324 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.698325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.698415 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.698776 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:26.698783 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:26.699999 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:26.701512 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:26.701518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:26.701520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:26.701725 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:26.701936 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:26.701936 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:26.701937 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:26.702036 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:26.702308 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:26.702313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:26.702436 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 11:15:26.702536 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 11:15:26.702675 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260927 11:15:26.703274 14594 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38695
I20260927 11:15:26.703359 14594 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:26.703521 14594 heartbeater.cc:511] Master 127.0.98.62:38695 requested a full tablet report, sending...
I20260927 11:15:26.703871 14135 ts_manager.cc:194] Registered new tserver with Master: b31be1cacc454e18a5cc67d33cab6c61 (127.0.98.3:45691)
I20260927 11:15:26.704227 14135 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38891
I20260927 11:15:26.708474 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:15:26.715602 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.709774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.0.98.62:38695)
0927 11:15:26.709833 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:26.709835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:26.709944 (+ 109us) server_negotiation.cc:407] Connection header received
0927 11:15:26.710015 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:26.710016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:26.710025 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:26.710101 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:26.710654 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.711099 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.711813 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.712004 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.712070 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.713188 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:26.713190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:26.713190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:26.713196 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:26.714514 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:26.714846 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:26.714848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:26.714848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:26.714897 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:26.715089 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:26.715090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:26.715090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:26.715225 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:26.715318 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:26.715440 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 11:15:26.715509 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 11:15:26.717127 14135 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:15:26.718780 14135 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:15:26.724691 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.720559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45691 (local address 127.0.0.1:54234)
0927 11:15:26.720809 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:26.720849 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:26.720912 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:26.721342 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:26.721343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:26.721345 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:26.721397 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.721398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.722455 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.722456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.723715 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.723717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.724418 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.724419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.724491 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.724504 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 11:15:26.724565 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 11:15:26.724610 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":4,"thread_start_us":69,"threads_started":1}
I20260927 11:15:26.724797 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.720808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36528 (local address 127.0.98.1:43815)
0927 11:15:26.720993 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:26.720998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:26.721007 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:15:26.721051 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:26.721053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:26.721065 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:26.721148 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:26.721542 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.722702 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.723805 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.724461 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.724526 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.724575 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:26.724668 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 11:15:26.724717 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260927 11:15:26.724792 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.720559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43815 (local address 127.0.0.1:36528)
0927 11:15:26.720809 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:26.720848 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:26.720906 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:26.721242 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:26.721245 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:26.721247 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:26.721307 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.721309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.722831 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.722832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.723729 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.723731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.724538 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.724539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.724592 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.724594 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:15:26.724655 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 11:15:26.724685 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260927 11:15:26.725046 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.720574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40749 (local address 127.0.0.1:53174)
0927 11:15:26.721009 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:26.721017 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:26.721057 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:26.721422 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:26.721423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:26.721425 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:26.721468 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.721469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.722770 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.722771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.723931 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.723933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.724838 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.724839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.724889 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.724892 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:15:26.724937 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 11:15:26.724973 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":24,"thread_start_us":90,"threads_started":1}
I20260927 11:15:26.725098 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.721014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53174 (local address 127.0.98.2:40749)
0927 11:15:26.721197 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:26.721202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:26.721211 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:15:26.721247 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:26.721249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:26.721263 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:26.721339 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:26.721546 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.722669 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.724009 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.724685 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.724754 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.724802 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:26.724950 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 11:15:26.724999 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260927 11:15:26.725368 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.720652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54234 (local address 127.0.98.3:45691)
0927 11:15:26.721210 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:26.721213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:26.721220 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:15:26.721247 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:26.721248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:26.721258 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:26.721342 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:26.721456 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.722283 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.723803 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.724345 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.724418 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.724473 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:26.725246 (+ 773us) server_negotiation.cc:299] Negotiation successful
0927 11:15:26.725299 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":88,"threads_started":1}
I20260927 11:15:26.725724 14260 tablet_service.cc:1467] Processing CreateTablet for tablet 74e9b6742a12464083ffce54f7ca031e (DEFAULT_TABLE table=test-table [id=feb87321d65d4d749ee28063bb02a9e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:15:26.725739 14394 tablet_service.cc:1467] Processing CreateTablet for tablet 74e9b6742a12464083ffce54f7ca031e (DEFAULT_TABLE table=test-table [id=feb87321d65d4d749ee28063bb02a9e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:15:26.726120 14394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e9b6742a12464083ffce54f7ca031e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:26.726137 14260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e9b6742a12464083ffce54f7ca031e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:26.728343 14608 tablet_bootstrap.cc:492] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: Bootstrap starting.
I20260927 11:15:26.728643 14609 tablet_bootstrap.cc:492] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: Bootstrap starting.
I20260927 11:15:26.728717 14528 tablet_service.cc:1467] Processing CreateTablet for tablet 74e9b6742a12464083ffce54f7ca031e (DEFAULT_TABLE table=test-table [id=feb87321d65d4d749ee28063bb02a9e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:15:26.728984 14528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74e9b6742a12464083ffce54f7ca031e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:26.729048 14608 tablet_bootstrap.cc:654] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:26.729218 14609 tablet_bootstrap.cc:654] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:26.729593 14609 log.cc:824] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:26.729578 14608 log.cc:824] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:26.730338 14609 tablet_bootstrap.cc:492] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: No bootstrap required, opened a new log
I20260927 11:15:26.730418 14609 ts_tablet_manager.cc:1397] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:15:26.730545 14608 tablet_bootstrap.cc:492] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: No bootstrap required, opened a new log
I20260927 11:15:26.730615 14608 ts_tablet_manager.cc:1397] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:15:26.732201 14609 raft_consensus.cc:348] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } } peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } }
I20260927 11:15:26.732378 14612 tablet_bootstrap.cc:492] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61: Bootstrap starting.
I20260927 11:15:26.732463 14609 raft_consensus.cc:374] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:26.732517 14609 raft_consensus.cc:729] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04011c612004fabbdd89cefc40e4083, State: Initialized, Role: FOLLOWER
I20260927 11:15:26.732642 14609 consensus_queue.cc:260] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } } peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } }
I20260927 11:15:26.732924 14609 ts_tablet_manager.cc:1428] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 11:15:26.732993 14612 tablet_bootstrap.cc:654] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:26.732924 14608 raft_consensus.cc:348] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } } peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } }
I20260927 11:15:26.733103 14608 raft_consensus.cc:374] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:26.733141 14460 heartbeater.cc:503] Master 127.0.98.62:38695 was elected leader, sending a full tablet report...
I20260927 11:15:26.733148 14608 raft_consensus.cc:729] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220afa49dff44ded9274015f228645f8, State: Initialized, Role: FOLLOWER
I20260927 11:15:26.733250 14608 consensus_queue.cc:260] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } } peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } }
I20260927 11:15:26.733501 14608 ts_tablet_manager.cc:1428] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:15:26.733531 14612 log.cc:824] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:26.733536 14326 heartbeater.cc:503] Master 127.0.98.62:38695 was elected leader, sending a full tablet report...
I20260927 11:15:26.734459 14612 tablet_bootstrap.cc:492] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61: No bootstrap required, opened a new log
I20260927 11:15:26.734539 14612 ts_tablet_manager.cc:1397] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:15:26.735775 14612 raft_consensus.cc:348] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } } peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } }
I20260927 11:15:26.735890 14612 raft_consensus.cc:374] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:26.735930 14612 raft_consensus.cc:729] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b31be1cacc454e18a5cc67d33cab6c61, State: Initialized, Role: FOLLOWER
I20260927 11:15:26.736037 14612 consensus_queue.cc:260] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } } peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } }
I20260927 11:15:26.736290 14612 ts_tablet_manager.cc:1428] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:15:26.736316 14594 heartbeater.cc:503] Master 127.0.98.62:38695 was elected leader, sending a full tablet report...
W20260927 11:15:26.808799 14461 tablet.cc:2369] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:15:26.891723 14327 tablet.cc:2369] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:15:26.944459 14595 tablet.cc:2369] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:15:26.969367 14613 raft_consensus.cc:482] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:15:26.969485 14613 raft_consensus.cc:504] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } } peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } }
I20260927 11:15:26.969833 14613 leader_election.cc:288] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 220afa49dff44ded9274015f228645f8 (127.0.98.1:43815), b31be1cacc454e18a5cc67d33cab6c61 (127.0.98.3:45691)
I20260927 11:15:26.972743 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.970104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43645 (local address 127.0.98.1:43815)
0927 11:15:26.970155 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:26.970157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:26.970174 (+ 17us) server_negotiation.cc:407] Connection header received
0927 11:15:26.970221 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:26.970222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:26.970230 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:26.970297 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:26.970495 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.971084 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.972020 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.972380 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.972454 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.972464 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:26.972647 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 11:15:26.972684 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 11:15:26.972746 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.970024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43815 (local address 127.0.98.2:43645)
0927 11:15:26.970088 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:26.970099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:26.970144 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:26.970319 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:26.970320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:26.970322 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:26.970411 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.970412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.971159 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.971160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.971928 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.971929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.972454 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.972455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.972553 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.972557 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:15:26.972615 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 11:15:26.972678 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":2}
I20260927 11:15:26.972841 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.970095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47407 (local address 127.0.98.3:45691)
0927 11:15:26.970153 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:26.970155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:26.970487 (+ 332us) server_negotiation.cc:407] Connection header received
0927 11:15:26.970560 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:26.970561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:26.970571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:26.970610 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:26.970754 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.971188 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.972101 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.972463 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.972553 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.972562 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:26.972723 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 11:15:26.972759 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:15:26.972867 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:26.970024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45691 (local address 127.0.98.2:47407)
0927 11:15:26.970416 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:26.970425 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:26.970500 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:26.970627 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:26.970628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:26.970630 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:26.970697 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.970698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.971268 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.971269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.972015 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:26.972017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:26.972567 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:26.972568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:26.972650 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:26.972653 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:15:26.972706 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 11:15:26.972744 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":192,"thread_start_us":114,"threads_started":1}
I20260927 11:15:26.973205 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e9b6742a12464083ffce54f7ca031e" candidate_uuid: "e04011c612004fabbdd89cefc40e4083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31be1cacc454e18a5cc67d33cab6c61" is_pre_election: true
I20260927 11:15:26.973263 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e9b6742a12464083ffce54f7ca031e" candidate_uuid: "e04011c612004fabbdd89cefc40e4083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220afa49dff44ded9274015f228645f8" is_pre_election: true
I20260927 11:15:26.973353 14548 raft_consensus.cc:2454] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e04011c612004fabbdd89cefc40e4083 in term 0.
I20260927 11:15:26.973424 14280 raft_consensus.cc:2454] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e04011c612004fabbdd89cefc40e4083 in term 0.
I20260927 11:15:26.973582 14348 leader_election.cc:302] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b31be1cacc454e18a5cc67d33cab6c61, e04011c612004fabbdd89cefc40e4083; no voters:
I20260927 11:15:26.973704 14613 raft_consensus.cc:2781] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:15:26.973765 14613 raft_consensus.cc:482] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:15:26.973783 14613 raft_consensus.cc:3037] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:26.974563 14613 raft_consensus.cc:504] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } } peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } }
I20260927 11:15:26.974697 14613 leader_election.cc:288] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [CANDIDATE]: Term 1 election: Requested vote from peers 220afa49dff44ded9274015f228645f8 (127.0.98.1:43815), b31be1cacc454e18a5cc67d33cab6c61 (127.0.98.3:45691)
I20260927 11:15:26.974852 14280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e9b6742a12464083ffce54f7ca031e" candidate_uuid: "e04011c612004fabbdd89cefc40e4083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220afa49dff44ded9274015f228645f8"
I20260927 11:15:26.974915 14280 raft_consensus.cc:3037] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:26.975582 14280 raft_consensus.cc:2454] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e04011c612004fabbdd89cefc40e4083 in term 1.
I20260927 11:15:26.975689 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74e9b6742a12464083ffce54f7ca031e" candidate_uuid: "e04011c612004fabbdd89cefc40e4083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31be1cacc454e18a5cc67d33cab6c61"
I20260927 11:15:26.975736 14347 leader_election.cc:302] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220afa49dff44ded9274015f228645f8, e04011c612004fabbdd89cefc40e4083; no voters:
I20260927 11:15:26.975756 14548 raft_consensus.cc:3037] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:26.975837 14613 raft_consensus.cc:2781] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:15:26.975977 14613 raft_consensus.cc:686] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 1 LEADER]: Becoming Leader. State: Replica: e04011c612004fabbdd89cefc40e4083, State: Running, Role: LEADER
I20260927 11:15:26.976068 14613 consensus_queue.cc:237] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } } peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } }
I20260927 11:15:26.976612 14548 raft_consensus.cc:2454] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e04011c612004fabbdd89cefc40e4083 in term 1.
I20260927 11:15:26.976821 14135 catalog_manager.cc:5495] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 reported cstate change: term changed from 0 to 1, leader changed from <none> to e04011c612004fabbdd89cefc40e4083 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e04011c612004fabbdd89cefc40e4083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } health_report { overall_health: HEALTHY } } }
I20260927 11:15:27.004281 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.002009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.0.98.2:40749)
0927 11:15:27.002066 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:27.002068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:27.002077 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:15:27.002146 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:27.002147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:27.002173 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:27.002245 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:15:27.002367 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.002798 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.003527 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.003709 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.003789 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.003947 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:15:27.004040 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:15:27.004119 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:27.004146 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 11:15:27.004173 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:15:27.005749 14548 raft_consensus.cc:1261] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61 [term 1 FOLLOWER]: Refusing update from remote peer e04011c612004fabbdd89cefc40e4083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:15:27.005767 14280 raft_consensus.cc:1261] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Refusing update from remote peer e04011c612004fabbdd89cefc40e4083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:15:27.006081 14618 consensus_queue.cc:1035] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:15:27.006198 14613 consensus_queue.cc:1035] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:15:27.008288 14621 mvcc.cc:205] Tried to move back new op lower bound from 7333919649813417984 to 7333919649694625792. Current Snapshot: MvccSnapshot[applied={T|T < 7333919649813417984}]
I20260927 11:15:27.008421 14623 mvcc.cc:205] Tried to move back new op lower bound from 7333919649813417984 to 7333919649694625792. Current Snapshot: MvccSnapshot[applied={T|T < 7333919649813417984}]
I20260927 11:15:27.009152 14622 mvcc.cc:205] Tried to move back new op lower bound from 7333919649813417984 to 7333919649694625792. Current Snapshot: MvccSnapshot[applied={T|T < 7333919649813417984 or (T in {7333919649813417984})}]
I20260927 11:15:27.010234 14135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:15:27.010311 14135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:15:27.010977 14135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: Sending DeleteTablet for 3 replicas of tablet 74e9b6742a12464083ffce54f7ca031e
I20260927 11:15:27.011300 14528 tablet_service.cc:1514] Processing DeleteTablet for tablet 74e9b6742a12464083ffce54f7ca031e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:15:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54234
I20260927 11:15:27.011300 14394 tablet_service.cc:1514] Processing DeleteTablet for tablet 74e9b6742a12464083ffce54f7ca031e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:15:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53174
I20260927 11:15:27.011320 14260 tablet_service.cc:1514] Processing DeleteTablet for tablet 74e9b6742a12464083ffce54f7ca031e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:15:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36528
I20260927 11:15:27.011518 14633 tablet_replica.cc:330] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: stopping tablet replica
I20260927 11:15:27.011581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 14197
I20260927 11:15:27.011610 14632 tablet_replica.cc:330] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61: stopping tablet replica
I20260927 11:15:27.011678 14632 raft_consensus.cc:2229] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:15:27.011751 14632 raft_consensus.cc:2258] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:15:27.012091 14632 ts_tablet_manager.cc:1905] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:15:27.013585 14632 ts_tablet_manager.cc:1918] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:15:27.013631 14632 log.cc:1196] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/74e9b6742a12464083ffce54f7ca031e
I20260927 11:15:27.013820 14632 ts_tablet_manager.cc:1939] T 74e9b6742a12464083ffce54f7ca031e P b31be1cacc454e18a5cc67d33cab6c61: Deleting consensus metadata
I20260927 11:15:27.014143 14120 catalog_manager.cc:4841] TS b31be1cacc454e18a5cc67d33cab6c61 (127.0.98.3:45691): tablet 74e9b6742a12464083ffce54f7ca031e (table test-table [id=feb87321d65d4d749ee28063bb02a9e9]) successfully deleted
W20260927 11:15:27.017916 14119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43815 (error 108)
I20260927 11:15:27.018007 14631 tablet_replica.cc:330] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: stopping tablet replica
I20260927 11:15:27.018172 14631 raft_consensus.cc:2229] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 1 LEADER]: Raft consensus shutting down.
W20260927 11:15:27.018328 14602 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:15:27.018212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43815 (local address 127.0.0.1:36542)
0927 11:15:27.018258 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:27.018296 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43815: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 11:15:27.018442 14119 catalog_manager.cc:4584] TS 220afa49dff44ded9274015f228645f8 (127.0.98.1:43815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 74e9b6742a12464083ffce54f7ca031e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43815: connect: Connection reset by peer (error 104)
I20260927 11:15:27.018477 14631 raft_consensus.cc:2258] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:15:27.018702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 14331
I20260927 11:15:27.023485 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.023377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40749 (local address 127.0.0.1:53192)
0927 11:15:27.023427 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:27.023453 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 11:15:27.023587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 14465
I20260927 11:15:27.028331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 14103
I20260927 11:15:27.033560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38695
--webserver_interface=127.0.98.62
--webserver_port=44969
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:27.110424 14635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:27.110618 14635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:27.110641 14635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:27.111703 14635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:15:27.111737 14635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:27.111752 14635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:27.111764 14635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:15:27.111776 14635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:15:27.113103 14635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:27.113269 14635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:27.113417 14635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:27.121477 14644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:27.121531 14642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:27.121491 14641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:27.121749 14635 server_base.cc:1034] running on GCE node
I20260927 11:15:27.122035 14635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:27.122368 14635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:27.123517 14635 hybrid_clock.cc:648] HybridClock initialized: now 1790507727123511 us; error 33 us; skew 500 ppm
Sep 27 11:15:27 dist-test-slave-9hq9 krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:27.126219 14635 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:15:27.126688 14635 webserver.cc:460] Webserver started at http://127.0.98.62:44969/ using document root <none> and password file <none>
I20260927 11:15:27.126910 14635 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:27.126981 14635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:27.128270 14635 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:15:27.128865 14651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:27.129021 14635 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:27.129114 14635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "37a2d40778c94662958cac084069641e"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "8bed7fe6207f7d154899ba11e56bc9ed"
server_key_iv: "6307da24148db027e3ce6b1d5d0aa3de"
server_key_version: "encryptionkey@0"
I20260927 11:15:27.129478 14635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:27.143739 14635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:27.145354 14635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:27.145494 14635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:27.149233 14635 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38695
I20260927 11:15:27.149276 14703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38695 every 8 connection(s)
I20260927 11:15:27.149627 14635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:15:27.151208 14704 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:15:27.152086 14704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: Bootstrap starting.
I20260927 11:15:27.158318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 14635
I20260927 11:15:27.158595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43815
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38695
--webserver_port=46221
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:15:27.160091 14704 log.cc:824] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:27.161863 14704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:15:27.162182 14704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: Bootstrap complete.
I20260927 11:15:27.164368 14704 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } }
I20260927 11:15:27.164598 14704 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a2d40778c94662958cac084069641e, State: Initialized, Role: FOLLOWER
I20260927 11:15:27.164774 14704 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } }
I20260927 11:15:27.164852 14704 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:15:27.164906 14704 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:15:27.164951 14704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 1 FOLLOWER]: Advancing to term 2
I20260927 11:15:27.165875 14704 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } }
I20260927 11:15:27.165980 14704 leader_election.cc:302] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37a2d40778c94662958cac084069641e; no voters:
I20260927 11:15:27.166138 14704 leader_election.cc:288] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 11:15:27.166194 14708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 2 FOLLOWER]: Leader election won for term 2
I20260927 11:15:27.166361 14708 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 2 LEADER]: Becoming Leader. State: Replica: 37a2d40778c94662958cac084069641e, State: Running, Role: LEADER
I20260927 11:15:27.166394 14704 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:15:27.166514 14708 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } }
I20260927 11:15:27.166916 14710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37a2d40778c94662958cac084069641e. Latest consensus state: current_term: 2 leader_uuid: "37a2d40778c94662958cac084069641e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } } }
I20260927 11:15:27.167002 14710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:27.167379 14709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "37a2d40778c94662958cac084069641e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } } }
I20260927 11:15:27.167459 14709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:27.167574 14715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:15:27.168262 14715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:15:27.168371 14715 catalog_manager.cc:1259] Loaded cluster ID: 8551c1fc6c8d4fbc961853a03fdde03e
I20260927 11:15:27.168413 14715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:15:27.168686 14715 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 11:15:27.172565 14715 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 11:15:27.174827 14715 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:15:27.175015 14715 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: Loaded TSK: 0
I20260927 11:15:27.175237 14715 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 11:15:27.237607 14706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:27.237809 14706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:27.237844 14706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:27.238986 14706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:27.239045 14706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:27.239090 14706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:15:27.240459 14706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46221
--tserver_master_addrs=127.0.98.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:27.240691 14706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:27.240916 14706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:27.245762 14734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:27.245724 14732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:27.245971 14706 server_base.cc:1034] running on GCE node
W20260927 11:15:27.245757 14731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:27.246251 14706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:27.246469 14706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:27.247617 14706 hybrid_clock.cc:648] HybridClock initialized: now 1790507727247604 us; error 30 us; skew 500 ppm
Sep 27 11:15:27 dist-test-slave-9hq9 krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:27.250002 14706 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:15:27.250445 14706 webserver.cc:460] Webserver started at http://127.0.98.1:46221/ using document root <none> and password file <none>
I20260927 11:15:27.250669 14706 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:27.250743 14706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:27.252004 14706 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:15:27.252673 14741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:27.252813 14706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:15:27.252902 14706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "220afa49dff44ded9274015f228645f8"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "90e5e6b06cf5b124c32b4b3807f9a8e3"
server_key_iv: "dbe0b28883742441a6e4b311198f9609"
server_key_version: "encryptionkey@0"
I20260927 11:15:27.253191 14706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:27.265657 14706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:27.268033 14706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:27.268221 14706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:27.268433 14706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:27.268850 14748 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 11:15:27.270602 14706 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 11:15:27.270658 14706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 11:15:27.270696 14706 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 11:15:27.271234 14706 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 11:15:27.271283 14706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 11:15:27.271346 14748 tablet_bootstrap.cc:492] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: Bootstrap starting.
I20260927 11:15:27.278155 14706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43815
I20260927 11:15:27.278478 14706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:15:27.281900 14748 log.cc:824] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:27.282168 14855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43815 every 8 connection(s)
I20260927 11:15:27.283028 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 14706
I20260927 11:15:27.283342 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40749
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38695
--webserver_port=34873
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:15:27.284150 14748 tablet_bootstrap.cc:492] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:15:27.284425 14748 tablet_bootstrap.cc:492] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: Bootstrap complete.
I20260927 11:15:27.285490 14748 ts_tablet_manager.cc:1397] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260927 11:15:27.286748 14748 raft_consensus.cc:348] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } } peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } }
I20260927 11:15:27.287027 14748 raft_consensus.cc:729] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220afa49dff44ded9274015f228645f8, State: Initialized, Role: FOLLOWER
I20260927 11:15:27.287339 14748 consensus_queue.cc:260] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } } peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } }
I20260927 11:15:27.287647 14748 ts_tablet_manager.cc:1428] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
Sep 27 11:15:27 dist-test-slave-9hq9 krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:27.294400 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.279156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45705 (local address 127.0.98.62:38695)
0927 11:15:27.282469 (+ 3313us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:27.282478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:27.282513 (+ 35us) server_negotiation.cc:407] Connection header received
0927 11:15:27.282753 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:27.282758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:27.282804 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:27.282953 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:27.284220 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.285267 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.286775 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.287378 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.287454 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.290300 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:27.290304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:27.290305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:27.290316 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:27.292305 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:27.292851 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:27.292854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:27.292855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:27.293000 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:27.293325 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:27.293326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:27.293326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:27.293605 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:27.293688 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:27.294074 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 11:15:27.294212 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3267,"thread_start_us":74,"threads_started":1}
I20260927 11:15:27.294407 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.279105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:45705)
0927 11:15:27.281890 (+ 2785us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:27.281921 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:27.282532 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:27.283181 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:27.283185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:27.283789 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:27.284070 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.284075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.285369 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.285370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.286678 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.286680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.287585 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.287585 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.287736 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.288317 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:27.288328 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:27.290180 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:27.292414 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:27.292422 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:27.292424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:27.292739 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:27.293094 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:27.293095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:27.293095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:27.293234 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:27.293681 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:27.293687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:27.293849 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 11:15:27.293962 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 11:15:27.294160 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2750,"thread_start_us":95,"threads_started":1}
I20260927 11:15:27.295358 14856 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38695
I20260927 11:15:27.295446 14856 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:27.295610 14856 heartbeater.cc:511] Master 127.0.98.62:38695 requested a full tablet report, sending...
I20260927 11:15:27.296265 14668 ts_manager.cc:194] Registered new tserver with Master: 220afa49dff44ded9274015f228645f8 (127.0.98.1:43815)
I20260927 11:15:27.297318 14668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45705
I20260927 11:15:27.298183 14856 heartbeater.cc:503] Master 127.0.98.62:38695 was elected leader, sending a full tablet report...
I20260927 11:15:27.305368 14864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.296962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36554 (local address 127.0.98.1:43815)
0927 11:15:27.302204 (+ 5242us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:27.302208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:27.302263 (+ 55us) server_negotiation.cc:407] Connection header received
0927 11:15:27.302375 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:27.302377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:27.302393 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:27.302471 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:27.302601 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.303200 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.304593 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.305020 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.305072 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.305122 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:27.305300 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 11:15:27.305325 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5193,"thread_start_us":50,"threads_started":1}
I20260927 11:15:27.305444 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.296779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43815 (local address 127.0.0.1:36554)
0927 11:15:27.302161 (+ 5382us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:27.302213 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:27.302283 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:27.302485 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:27.302487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:27.302489 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:27.302542 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.302543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.303264 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.303264 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.304505 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.304508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.305106 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.305107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.305186 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.305198 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 11:15:27.305282 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 11:15:27.305331 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5353,"thread_start_us":64,"threads_started":1}
I20260927 11:15:27.305778 14790 tablet_service.cc:1514] Processing DeleteTablet for tablet 74e9b6742a12464083ffce54f7ca031e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:15:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36554
I20260927 11:15:27.306342 14865 tablet_replica.cc:330] stopping tablet replica
I20260927 11:15:27.306449 14865 raft_consensus.cc:2229] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:15:27.306519 14865 raft_consensus.cc:2258] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:15:27.306772 14865 ts_tablet_manager.cc:1905] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:15:27.308635 14865 ts_tablet_manager.cc:1918] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:15:27.308741 14865 log.cc:1196] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal/wals/74e9b6742a12464083ffce54f7ca031e
I20260927 11:15:27.309108 14865 ts_tablet_manager.cc:1939] T 74e9b6742a12464083ffce54f7ca031e P 220afa49dff44ded9274015f228645f8: Deleting consensus metadata
I20260927 11:15:27.309554 14652 catalog_manager.cc:4841] TS 220afa49dff44ded9274015f228645f8 (127.0.98.1:43815): tablet 74e9b6742a12464083ffce54f7ca031e (table test-table [id=feb87321d65d4d749ee28063bb02a9e9]) successfully deleted
W20260927 11:15:27.366293 14861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:27.366454 14861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:27.366477 14861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:27.367648 14861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:27.367715 14861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:27.367743 14861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:15:27.369072 14861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34873
--tserver_master_addrs=127.0.98.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:27.369237 14861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:27.369405 14861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:27.373962 14873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:27.373963 14870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:27.374003 14871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:27.374082 14861 server_base.cc:1034] running on GCE node
I20260927 11:15:27.374351 14861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:27.374601 14861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:27.375747 14861 hybrid_clock.cc:648] HybridClock initialized: now 1790507727375731 us; error 26 us; skew 500 ppm
Sep 27 11:15:27 dist-test-slave-9hq9 krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:27.377964 14861 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:15:27.378357 14861 webserver.cc:460] Webserver started at http://127.0.98.2:34873/ using document root <none> and password file <none>
I20260927 11:15:27.378571 14861 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:27.378650 14861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:27.379837 14861 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:15:27.380410 14880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:27.380561 14861 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:15:27.380654 14861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "e04011c612004fabbdd89cefc40e4083"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "2a19acea7d8b405a1890b79b94084975"
server_key_iv: "4118f28a9f6b3fa575952cd5704bfd69"
server_key_version: "encryptionkey@0"
I20260927 11:15:27.380930 14861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:27.389748 14861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:27.391362 14861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:27.391495 14861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:27.391680 14861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:27.392132 14887 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 11:15:27.393337 14861 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 11:15:27.393386 14861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260927 11:15:27.393435 14861 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 11:15:27.393973 14861 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 11:15:27.394022 14861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 11:15:27.394083 14887 tablet_bootstrap.cc:492] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: Bootstrap starting.
I20260927 11:15:27.395627 14887 log.cc:824] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:27.398340 14887 tablet_bootstrap.cc:492] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:15:27.398610 14887 tablet_bootstrap.cc:492] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: Bootstrap complete.
I20260927 11:15:27.398759 14887 ts_tablet_manager.cc:1397] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 11:15:27.400113 14887 raft_consensus.cc:348] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } } peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } }
I20260927 11:15:27.400380 14887 raft_consensus.cc:729] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04011c612004fabbdd89cefc40e4083, State: Initialized, Role: FOLLOWER
I20260927 11:15:27.400563 14887 consensus_queue.cc:260] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } } peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } }
I20260927 11:15:27.400642 14861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40749
I20260927 11:15:27.400679 14995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40749 every 8 connection(s)
I20260927 11:15:27.400833 14887 ts_tablet_manager.cc:1428] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:15:27.400997 14861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:15:27 dist-test-slave-9hq9 krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:27.408859 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.401603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37745 (local address 127.0.98.62:38695)
0927 11:15:27.401690 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:27.401692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:27.401833 (+ 141us) server_negotiation.cc:407] Connection header received
0927 11:15:27.401986 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:27.401987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:27.401999 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:27.402081 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:27.402656 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.403174 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.404136 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.404358 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.404438 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.406666 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:27.406667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:27.406667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:27.406673 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:27.407779 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:27.408105 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:27.408106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:27.408106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:27.408155 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:27.408356 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:27.408357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:27.408357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:27.408447 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:27.408494 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:27.408697 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 11:15:27.408790 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 11:15:27.408986 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.401564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:37745)
0927 11:15:27.401713 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:27.401748 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:27.401900 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:27.402142 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:27.402145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:27.402383 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:27.402549 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.402552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.403297 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.403298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.404044 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.404046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.404446 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.404447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.404574 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.404951 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:27.404958 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:27.406585 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:27.407849 (+ 1264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:27.407854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:27.407856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:27.408051 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:27.408204 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:27.408205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:27.408205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:27.408286 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:27.408495 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:27.408498 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:27.408602 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 11:15:27.408675 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 11:15:27.408818 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260927 11:15:27.409209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 14861
I20260927 11:15:27.409420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45691
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38695
--webserver_port=46549
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:15:27.411042 14997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38695
I20260927 11:15:27.411123 14997 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:27.411286 14997 heartbeater.cc:511] Master 127.0.98.62:38695 requested a full tablet report, sending...
I20260927 11:15:27.411741 14668 ts_manager.cc:194] Registered new tserver with Master: e04011c612004fabbdd89cefc40e4083 (127.0.98.2:40749)
I20260927 11:15:27.412292 14668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37745
Sep 27 11:15:27 dist-test-slave-9hq9 krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 11:15:27.419799 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.412065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53198 (local address 127.0.98.2:40749)
0927 11:15:27.412219 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:27.412223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:27.412233 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:15:27.412261 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:27.412263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:27.412275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:27.412348 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:27.412906 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.413575 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.414650 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.414917 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.414970 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.417059 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:27.417064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:27.417065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:27.417071 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:27.418644 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:27.419082 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:27.419084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:27.419084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:27.419131 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:27.419322 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:27.419322 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:27.419323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:27.419429 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:27.419484 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:27.419658 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 11:15:27.419722 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260927 11:15:27.419934 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.412009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40749 (local address 127.0.0.1:53198)
0927 11:15:27.412050 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:27.412061 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:27.412107 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:27.412368 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:27.412369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:27.412725 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:27.412764 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.412767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.413661 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.413662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.414575 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.414578 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.414988 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.414991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.415056 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.415467 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:27.415474 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:27.416938 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:27.418722 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:27.418729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:27.418731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:27.418999 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:27.419189 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:27.419191 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:27.419191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:27.419261 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:27.419490 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:27.419496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:27.419553 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 11:15:27.419634 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 11:15:27.419751 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 11:15:27.420437 14930 tablet_service.cc:1514] Processing DeleteTablet for tablet 74e9b6742a12464083ffce54f7ca031e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:15:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53198
I20260927 11:15:27.420960 15002 tablet_replica.cc:330] stopping tablet replica
I20260927 11:15:27.421054 15002 raft_consensus.cc:2229] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:15:27.421128 15002 raft_consensus.cc:2258] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:15:27.421379 15002 ts_tablet_manager.cc:1905] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:15:27.423588 15002 ts_tablet_manager.cc:1918] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:15:27.423658 15002 log.cc:1196] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/74e9b6742a12464083ffce54f7ca031e
I20260927 11:15:27.423887 15002 ts_tablet_manager.cc:1939] T 74e9b6742a12464083ffce54f7ca031e P e04011c612004fabbdd89cefc40e4083: Deleting consensus metadata
I20260927 11:15:27.424331 14654 catalog_manager.cc:4841] TS e04011c612004fabbdd89cefc40e4083 (127.0.98.2:40749): tablet 74e9b6742a12464083ffce54f7ca031e (table test-table [id=feb87321d65d4d749ee28063bb02a9e9]) successfully deleted
W20260927 11:15:27.489169 15000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:27.489351 15000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:27.489373 15000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:27.490454 15000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:27.490494 15000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:27.490541 15000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:15:27.491899 15000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46549
--tserver_master_addrs=127.0.98.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:27.492100 15000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:27.492303 15000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:15:27.497044 15000 server_base.cc:1034] running on GCE node
W20260927 11:15:27.497009 15008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:27.497004 15010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:27.497008 15007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:27.497419 15000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:27.497658 15000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:27.498803 15000 hybrid_clock.cc:648] HybridClock initialized: now 1790507727498786 us; error 29 us; skew 500 ppm
Sep 27 11:15:27 dist-test-slave-9hq9 krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:27.501024 15000 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:15:27.501417 15000 webserver.cc:460] Webserver started at http://127.0.98.3:46549/ using document root <none> and password file <none>
I20260927 11:15:27.501627 15000 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:27.501701 15000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:27.502975 15000 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:15:27.503484 15017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:27.503638 15000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:15:27.503724 15000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "b31be1cacc454e18a5cc67d33cab6c61"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "3588a434c9b3fad7e5e8835ada0e327b"
server_key_iv: "d0b3b4711cc7ad0b94360f2724dfdd6d"
server_key_version: "encryptionkey@0"
I20260927 11:15:27.504005 15000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:27.510941 15000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:27.512956 15000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:27.513109 15000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:27.513381 15000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:27.513654 15000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:15:27.513706 15000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:27.513744 15000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:15:27.513773 15000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:27.519723 15000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45691
I20260927 11:15:27.519763 15130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45691 every 8 connection(s)
I20260927 11:15:27.520058 15000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:15:27.524186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 15000
Sep 27 11:15:27 dist-test-slave-9hq9 krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:27.528813 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.520707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35871 (local address 127.0.98.62:38695)
0927 11:15:27.520754 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:27.520756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:27.520949 (+ 193us) server_negotiation.cc:407] Connection header received
0927 11:15:27.521342 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:27.521343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:27.521353 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:27.521393 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:27.522215 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.522690 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.523657 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.523892 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.523948 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.525908 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:27.525909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:27.525909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:27.525916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:27.527486 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:27.527940 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:27.527941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:27.527941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:27.527993 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:27.528252 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:27.528253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:27.528253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:27.528342 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:27.528389 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:27.528675 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 11:15:27.528747 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:15:27.528992 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.520710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:35871)
0927 11:15:27.520849 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:27.520877 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:27.521260 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:27.521437 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:27.521440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:27.521923 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:27.522147 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.522151 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.522758 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.522758 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.523548 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.523550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.523961 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.523961 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.524120 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.524634 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:27.524643 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:27.525827 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:27.527549 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:27.527570 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:27.527572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:27.527862 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:27.528056 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:27.528056 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:27.528057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:27.528190 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:27.528402 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:27.528407 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:27.528553 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 11:15:27.528632 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 11:15:27.528766 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260927 11:15:27.529407 15131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38695
I20260927 11:15:27.529502 15131 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:27.529673 15131 heartbeater.cc:511] Master 127.0.98.62:38695 requested a full tablet report, sending...
I20260927 11:15:27.530018 14668 ts_manager.cc:194] Registered new tserver with Master: b31be1cacc454e18a5cc67d33cab6c61 (127.0.98.3:45691)
I20260927 11:15:27.530391 14668 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35871
I20260927 11:15:27.531308 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.524558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.0.98.62:38695)
0927 11:15:27.524681 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:27.524683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:27.524723 (+ 40us) server_negotiation.cc:407] Connection header received
0927 11:15:27.524798 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:27.524799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:27.524810 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:27.524881 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:27.525412 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.525939 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.526890 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.527195 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.527266 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.528507 (+ 1241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:27.528508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:27.528508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:27.528513 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:27.530222 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:27.530525 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:27.530526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:27.530526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:27.530579 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:27.530765 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:27.530765 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:27.530766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:27.530896 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:27.531050 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:27.531108 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 11:15:27.531204 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260927 11:15:27.532043 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:15:27.534718 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.532346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.0.98.62:38695)
0927 11:15:27.532383 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:27.532385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:27.532490 (+ 105us) server_negotiation.cc:407] Connection header received
0927 11:15:27.532568 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:27.532569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:27.532603 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:27.532672 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:15:27.532765 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.533214 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.533924 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.534187 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.534261 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.534439 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:15:27.534528 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:15:27.534613 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:27.534630 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 11:15:27.534650 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:15:27.534930 14668 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:15:27.536435 14668 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:15:27.538302 14930 tablet_service.cc:1467] Processing CreateTablet for tablet a953c78dfdeb4adaa22638f94e016ba9 (DEFAULT_TABLE table=test-table [id=f3c72a4ba74f44b58724dfd2a469ccaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:15:27.538476 14930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a953c78dfdeb4adaa22638f94e016ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:27.539964 14887 tablet_bootstrap.cc:492] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083: Bootstrap starting.
I20260927 11:15:27.540215 14790 tablet_service.cc:1467] Processing CreateTablet for tablet a953c78dfdeb4adaa22638f94e016ba9 (DEFAULT_TABLE table=test-table [id=f3c72a4ba74f44b58724dfd2a469ccaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:15:27.540357 14790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a953c78dfdeb4adaa22638f94e016ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:27.540371 14887 tablet_bootstrap.cc:654] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:27.541373 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.538203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54236 (local address 127.0.98.3:45691)
0927 11:15:27.538337 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:27.538341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:27.538350 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:15:27.538379 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:27.538384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:27.538396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:27.538487 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:27.538625 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.539546 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.540643 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.541092 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.541146 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.541178 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:27.541304 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 11:15:27.541326 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260927 11:15:27.541393 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.538010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45691 (local address 127.0.0.1:54236)
0927 11:15:27.538049 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:27.538079 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:27.538113 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:27.538488 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:27.538489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:27.538491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:27.538530 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.538531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.539674 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.539675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.540554 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.540557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.541172 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.541173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.541230 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.541232 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:15:27.541297 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 11:15:27.541332 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 11:15:27.541677 14748 tablet_bootstrap.cc:492] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8: Bootstrap starting.
I20260927 11:15:27.541970 14748 tablet_bootstrap.cc:654] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:27.541970 15065 tablet_service.cc:1467] Processing CreateTablet for tablet a953c78dfdeb4adaa22638f94e016ba9 (DEFAULT_TABLE table=test-table [id=f3c72a4ba74f44b58724dfd2a469ccaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:15:27.542097 14887 tablet_bootstrap.cc:492] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083: No bootstrap required, opened a new log
I20260927 11:15:27.542140 14887 ts_tablet_manager.cc:1397] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:15:27.542277 15065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a953c78dfdeb4adaa22638f94e016ba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:27.542279 14887 raft_consensus.cc:348] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } }
I20260927 11:15:27.542342 14887 raft_consensus.cc:374] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:27.542388 14887 raft_consensus.cc:729] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e04011c612004fabbdd89cefc40e4083, State: Initialized, Role: FOLLOWER
I20260927 11:15:27.542440 14887 consensus_queue.cc:260] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } }
I20260927 11:15:27.542501 14748 tablet_bootstrap.cc:492] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8: No bootstrap required, opened a new log
I20260927 11:15:27.542523 14887 ts_tablet_manager.cc:1428] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:27.542544 14748 ts_tablet_manager.cc:1397] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:15:27.542572 14997 heartbeater.cc:503] Master 127.0.98.62:38695 was elected leader, sending a full tablet report...
I20260927 11:15:27.542640 14748 raft_consensus.cc:348] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } }
I20260927 11:15:27.542709 14748 raft_consensus.cc:374] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:27.542757 14748 raft_consensus.cc:729] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220afa49dff44ded9274015f228645f8, State: Initialized, Role: FOLLOWER
I20260927 11:15:27.542804 14748 consensus_queue.cc:260] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } }
I20260927 11:15:27.542882 14748 ts_tablet_manager.cc:1428] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:15:27.545293 15140 tablet_bootstrap.cc:492] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61: Bootstrap starting.
I20260927 11:15:27.545831 15140 tablet_bootstrap.cc:654] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:27.546386 15140 log.cc:824] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:27.547031 15140 tablet_bootstrap.cc:492] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61: No bootstrap required, opened a new log
I20260927 11:15:27.547114 15140 ts_tablet_manager.cc:1397] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:15:27.548357 15140 raft_consensus.cc:348] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } }
I20260927 11:15:27.548502 15140 raft_consensus.cc:374] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:27.548545 15140 raft_consensus.cc:729] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b31be1cacc454e18a5cc67d33cab6c61, State: Initialized, Role: FOLLOWER
I20260927 11:15:27.548655 15140 consensus_queue.cc:260] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } }
I20260927 11:15:27.548897 15140 ts_tablet_manager.cc:1428] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:15:27.548938 15131 heartbeater.cc:503] Master 127.0.98.62:38695 was elected leader, sending a full tablet report...
W20260927 11:15:27.651676 14998 tablet.cc:2369] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:15:27.770581 15132 tablet.cc:2369] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:15:27.782707 14857 tablet.cc:2369] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:15:27.840626 14996 raft_consensus.cc:482] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:15:27.840740 14996 raft_consensus.cc:504] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } }
I20260927 11:15:27.841024 14996 leader_election.cc:288] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b31be1cacc454e18a5cc67d33cab6c61 (127.0.98.3:45691), 220afa49dff44ded9274015f228645f8 (127.0.98.1:43815)
I20260927 11:15:27.844813 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.841263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47125 (local address 127.0.98.3:45691)
0927 11:15:27.841322 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:27.841324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:27.841352 (+ 28us) server_negotiation.cc:407] Connection header received
0927 11:15:27.841427 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:27.841428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:27.841441 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:27.841519 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:27.841689 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.842458 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.843913 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.844479 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.844547 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.844557 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:27.844712 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 11:15:27.844748 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 11:15:27.844815 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.841290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34185 (local address 127.0.98.1:43815)
0927 11:15:27.841416 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:27.841418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:27.841501 (+ 83us) server_negotiation.cc:407] Connection header received
0927 11:15:27.841554 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:27.841555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:27.841566 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:27.841636 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:27.841761 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.842458 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.843913 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.844479 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.844547 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.844557 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:27.844712 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 11:15:27.844748 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260927 11:15:27.844849 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.841203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45691 (local address 127.0.98.2:47125)
0927 11:15:27.841259 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:27.841270 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:27.841325 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:27.841543 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:27.841544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:27.841546 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:27.841621 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.841623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.842539 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.842540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.843830 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.843834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.844554 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.844555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.844621 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.844644 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 11:15:27.844699 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 11:15:27.844741 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":50}
I20260927 11:15:27.844858 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.841212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43815 (local address 127.0.98.2:34185)
0927 11:15:27.841423 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:27.841432 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:27.841487 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:27.841648 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:27.841648 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:27.841649 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:27.841701 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.841702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.842528 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.842529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.843830 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:27.843834 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.844568 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.844568 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:27.844621 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.844644 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 11:15:27.844699 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 11:15:27.844741 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":60,"thread_start_us":129,"threads_started":1}
I20260927 11:15:27.845206 15085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a953c78dfdeb4adaa22638f94e016ba9" candidate_uuid: "e04011c612004fabbdd89cefc40e4083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31be1cacc454e18a5cc67d33cab6c61" is_pre_election: true
I20260927 11:15:27.845348 15085 raft_consensus.cc:2454] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e04011c612004fabbdd89cefc40e4083 in term 0.
I20260927 11:15:27.845317 14810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a953c78dfdeb4adaa22638f94e016ba9" candidate_uuid: "e04011c612004fabbdd89cefc40e4083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220afa49dff44ded9274015f228645f8" is_pre_election: true
I20260927 11:15:27.845460 14810 raft_consensus.cc:2454] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e04011c612004fabbdd89cefc40e4083 in term 0.
I20260927 11:15:27.845573 14882 leader_election.cc:302] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b31be1cacc454e18a5cc67d33cab6c61, e04011c612004fabbdd89cefc40e4083; no voters:
I20260927 11:15:27.845687 14996 raft_consensus.cc:2781] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:15:27.845742 14996 raft_consensus.cc:482] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:15:27.845757 14996 raft_consensus.cc:3037] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:27.846494 14996 raft_consensus.cc:504] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } }
I20260927 11:15:27.846627 14996 leader_election.cc:288] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [CANDIDATE]: Term 1 election: Requested vote from peers b31be1cacc454e18a5cc67d33cab6c61 (127.0.98.3:45691), 220afa49dff44ded9274015f228645f8 (127.0.98.1:43815)
I20260927 11:15:27.846738 15085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a953c78dfdeb4adaa22638f94e016ba9" candidate_uuid: "e04011c612004fabbdd89cefc40e4083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31be1cacc454e18a5cc67d33cab6c61"
I20260927 11:15:27.846786 14810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a953c78dfdeb4adaa22638f94e016ba9" candidate_uuid: "e04011c612004fabbdd89cefc40e4083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "220afa49dff44ded9274015f228645f8"
I20260927 11:15:27.846819 15085 raft_consensus.cc:3037] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:27.846870 14810 raft_consensus.cc:3037] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:27.847481 14810 raft_consensus.cc:2454] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e04011c612004fabbdd89cefc40e4083 in term 1.
I20260927 11:15:27.847549 15085 raft_consensus.cc:2454] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e04011c612004fabbdd89cefc40e4083 in term 1.
I20260927 11:15:27.847700 14881 leader_election.cc:302] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 220afa49dff44ded9274015f228645f8, e04011c612004fabbdd89cefc40e4083; no voters:
I20260927 11:15:27.847823 14996 raft_consensus.cc:2781] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:15:27.847956 14996 raft_consensus.cc:686] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [term 1 LEADER]: Becoming Leader. State: Replica: e04011c612004fabbdd89cefc40e4083, State: Running, Role: LEADER
I20260927 11:15:27.848031 14996 consensus_queue.cc:237] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } }
I20260927 11:15:27.848769 14668 catalog_manager.cc:5495] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 reported cstate change: term changed from 0 to 1, leader changed from <none> to e04011c612004fabbdd89cefc40e4083 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e04011c612004fabbdd89cefc40e4083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } health_report { overall_health: UNKNOWN } } }
I20260927 11:15:27.895565 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.893184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.0.98.2:40749)
0927 11:15:27.893228 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:27.893230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:27.893239 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:15:27.893301 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:27.893302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:27.893323 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:27.893392 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:15:27.893492 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.894017 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.894801 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:27.895002 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:27.895075 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:27.895238 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:15:27.895326 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:15:27.895400 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:27.895443 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 11:15:27.895462 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:15:27.896889 15085 raft_consensus.cc:1261] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61 [term 1 FOLLOWER]: Refusing update from remote peer e04011c612004fabbdd89cefc40e4083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:15:27.896889 14810 raft_consensus.cc:1261] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Refusing update from remote peer e04011c612004fabbdd89cefc40e4083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:15:27.897234 15145 consensus_queue.cc:1035] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:15:27.897369 14996 consensus_queue.cc:1035] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:15:27.899989 15148 mvcc.cc:205] Tried to move back new op lower bound from 7333919653463363584 to 7333919653266190336. Current Snapshot: MvccSnapshot[applied={T|T < 7333919653463363584}]
I20260927 11:15:27.900416 15149 mvcc.cc:205] Tried to move back new op lower bound from 7333919653463363584 to 7333919653266190336. Current Snapshot: MvccSnapshot[applied={T|T < 7333919653463363584 or (T in {7333919653463363584})}]
I20260927 11:15:27.901516 14667 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:15:27.901624 14667 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:15:27.902356 14667 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: Sending DeleteTablet for 3 replicas of tablet a953c78dfdeb4adaa22638f94e016ba9
I20260927 11:15:27.902642 14930 tablet_service.cc:1514] Processing DeleteTablet for tablet a953c78dfdeb4adaa22638f94e016ba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:15:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53198
I20260927 11:15:27.902704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 14706
I20260927 11:15:27.902745 15065 tablet_service.cc:1514] Processing DeleteTablet for tablet a953c78dfdeb4adaa22638f94e016ba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:15:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54236
I20260927 11:15:27.903155 15002 tablet_replica.cc:330] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083: stopping tablet replica
I20260927 11:15:27.903208 15002 raft_consensus.cc:2229] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:15:27.903277 15002 raft_consensus.cc:2258] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:15:27.903612 15002 ts_tablet_manager.cc:1905] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:15:27.905153 15002 ts_tablet_manager.cc:1918] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:15:27.905203 15002 log.cc:1196] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/a953c78dfdeb4adaa22638f94e016ba9
I20260927 11:15:27.905402 15002 ts_tablet_manager.cc:1939] T a953c78dfdeb4adaa22638f94e016ba9 P e04011c612004fabbdd89cefc40e4083: Deleting consensus metadata
I20260927 11:15:27.905719 14654 catalog_manager.cc:4841] TS e04011c612004fabbdd89cefc40e4083 (127.0.98.2:40749): tablet a953c78dfdeb4adaa22638f94e016ba9 (table test-table [id=f3c72a4ba74f44b58724dfd2a469ccaf]) successfully deleted
W20260927 11:15:27.908448 14652 connection.cc:535] client connection to 127.0.98.1:43815 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 11:15:27.908538 14652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 11:15:27.908723 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 14861
I20260927 11:15:27.908768 15157 tablet_replica.cc:330] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61: stopping tablet replica
I20260927 11:15:27.909361 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.908882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43815 (local address 127.0.0.1:36568)
0927 11:15:27.909283 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:27.909326 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382}
W20260927 11:15:27.910461 14652 catalog_manager.cc:4584] TS 220afa49dff44ded9274015f228645f8 (127.0.98.1:43815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a953c78dfdeb4adaa22638f94e016ba9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43815: connect: Connection refused (error 111)
I20260927 11:15:27.910915 15157 raft_consensus.cc:2229] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:15:27.911043 15157 raft_consensus.cc:2258] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:15:27.911404 15157 ts_tablet_manager.cc:1905] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:15:27.912813 15157 ts_tablet_manager.cc:1918] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:15:27.912869 15157 log.cc:1196] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/a953c78dfdeb4adaa22638f94e016ba9
I20260927 11:15:27.913255 15157 ts_tablet_manager.cc:1939] T a953c78dfdeb4adaa22638f94e016ba9 P b31be1cacc454e18a5cc67d33cab6c61: Deleting consensus metadata
I20260927 11:15:27.913345 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 15000
I20260927 11:15:27.917533 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 14635
I20260927 11:15:27.917536 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:27.917436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45691 (local address 127.0.0.1:54244)
0927 11:15:27.917480 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:27.917510 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:45691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 11:15:27.922370 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38695
--webserver_interface=127.0.98.62
--webserver_port=44969
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:27.999059 15159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:27.999240 15159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:27.999261 15159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:28.000300 15159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:15:28.000331 15159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:28.000344 15159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:28.000356 15159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:15:28.000368 15159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:15:28.001674 15159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:28.001838 15159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:28.001984 15159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:28.006296 15167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:28.006421 15165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:28.006335 15164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:28.006443 15159 server_base.cc:1034] running on GCE node
I20260927 11:15:28.006688 15159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:28.006902 15159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:28.008024 15159 hybrid_clock.cc:648] HybridClock initialized: now 1790507728008016 us; error 28 us; skew 500 ppm
Sep 27 11:15:28 dist-test-slave-9hq9 krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:28.010241 15159 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:15:28.010646 15159 webserver.cc:460] Webserver started at http://127.0.98.62:44969/ using document root <none> and password file <none>
I20260927 11:15:28.010802 15159 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:28.010838 15159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:28.011937 15159 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:15:28.012543 15174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:28.012686 15159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:15:28.012761 15159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "37a2d40778c94662958cac084069641e"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "8bed7fe6207f7d154899ba11e56bc9ed"
server_key_iv: "6307da24148db027e3ce6b1d5d0aa3de"
server_key_version: "encryptionkey@0"
I20260927 11:15:28.013015 15159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:28.023865 15159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:28.025708 15159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:28.025806 15159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:28.029345 15159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38695
I20260927 11:15:28.029381 15226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38695 every 8 connection(s)
I20260927 11:15:28.029731 15159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:15:28.031186 15227 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:15:28.031796 15227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: Bootstrap starting.
I20260927 11:15:28.036713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 15159
I20260927 11:15:28.036944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43815
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38695
--webserver_port=46221
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:15:28.040117 15227 log.cc:824] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:28.042136 15227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:15:28.042408 15227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e: Bootstrap complete.
I20260927 11:15:28.044466 15227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } }
I20260927 11:15:28.044687 15227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a2d40778c94662958cac084069641e, State: Initialized, Role: FOLLOWER
I20260927 11:15:28.044865 15227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } }
I20260927 11:15:28.044952 15227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:15:28.045002 15227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:15:28.045056 15227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 2 FOLLOWER]: Advancing to term 3
I20260927 11:15:28.045938 15227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } }
I20260927 11:15:28.046069 15227 leader_election.cc:302] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37a2d40778c94662958cac084069641e; no voters:
I20260927 11:15:28.046229 15227 leader_election.cc:288] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 11:15:28.046263 15231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 3 FOLLOWER]: Leader election won for term 3
I20260927 11:15:28.046371 15231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [term 3 LEADER]: Becoming Leader. State: Replica: 37a2d40778c94662958cac084069641e, State: Running, Role: LEADER
I20260927 11:15:28.046471 15227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:15:28.046505 15231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } }
I20260927 11:15:28.046849 15231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37a2d40778c94662958cac084069641e. Latest consensus state: current_term: 3 leader_uuid: "37a2d40778c94662958cac084069641e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } } }
I20260927 11:15:28.047096 15231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:28.047014 15232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "37a2d40778c94662958cac084069641e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a2d40778c94662958cac084069641e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38695 } } }
I20260927 11:15:28.047257 15232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a2d40778c94662958cac084069641e [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:28.047474 15240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:15:28.048269 15240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:15:28.048393 15240 catalog_manager.cc:1259] Loaded cluster ID: 8551c1fc6c8d4fbc961853a03fdde03e
I20260927 11:15:28.048458 15240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 11:15:28.048880 15240 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f85543d19ad google::LogMessage::Fail() at ??:0
@ 0x7f85543d488a google::LogMessage::SendToLog() at ??:0
@ 0x7f85543d14de google::LogMessage::Flush() at ??:0
@ 0x7f85543d2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f855c2f20f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f855c2fb8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f855629f0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8556294954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f85570f86db start_thread at ??:0
@ 0x7f8552ecf61f clone at ??:0
W20260927 11:15:28.120466 15229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:28.120668 15229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:28.120705 15229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:28.121915 15229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:28.121968 15229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:28.122017 15229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:15:28.123415 15229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46221
--tserver_master_addrs=127.0.98.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:28.123651 15229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:28.123855 15229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:15:28.128734 15229 server_base.cc:1034] running on GCE node
W20260927 11:15:28.128711 15264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:28.128815 15263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:28.128719 15266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:28.129140 15229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:28.129375 15229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:28.130523 15229 hybrid_clock.cc:648] HybridClock initialized: now 1790507728130508 us; error 31 us; skew 500 ppm
Sep 27 11:15:28 dist-test-slave-9hq9 krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:28.132824 15229 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:15:28.133229 15229 webserver.cc:460] Webserver started at http://127.0.98.1:46221/ using document root <none> and password file <none>
I20260927 11:15:28.133448 15229 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:28.133517 15229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:28.134824 15229 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260927 11:15:28.135473 15273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:28.135636 15229 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:15:28.135725 15229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "220afa49dff44ded9274015f228645f8"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "90e5e6b06cf5b124c32b4b3807f9a8e3"
server_key_iv: "dbe0b28883742441a6e4b311198f9609"
server_key_version: "encryptionkey@0"
I20260927 11:15:28.136065 15229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:28.144100 15229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:28.145924 15229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:28.146104 15229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:28.146312 15229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:28.146734 15280 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 11:15:28.148384 15229 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 11:15:28.148435 15229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 11:15:28.148475 15229 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 11:15:28.149101 15229 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 11:15:28.149150 15229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 11:15:28.149214 15280 tablet_bootstrap.cc:492] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8: Bootstrap starting.
I20260927 11:15:28.156255 15229 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43815
I20260927 11:15:28.156335 15387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43815 every 8 connection(s)
I20260927 11:15:28.156653 15229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:15:28.159667 15280 log.cc:824] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:28.161983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 15229
I20260927 11:15:28.162130 15280 tablet_bootstrap.cc:492] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:15:28.162315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40749
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38695
--webserver_port=34873
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:15:28.162431 15280 tablet_bootstrap.cc:492] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8: Bootstrap complete.
I20260927 11:15:28.163657 15280 ts_tablet_manager.cc:1397] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.011s
I20260927 11:15:28.165032 15280 raft_consensus.cc:348] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } }
I20260927 11:15:28.165355 15280 raft_consensus.cc:729] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 220afa49dff44ded9274015f228645f8, State: Initialized, Role: FOLLOWER
I20260927 11:15:28.165534 15280 consensus_queue.cc:260] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } }
I20260927 11:15:28.165791 15280 ts_tablet_manager.cc:1428] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
W20260927 11:15:28.239939 15390 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:15:28.157274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:48907)
0927 11:15:28.157447 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:28.157488 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:28.157924 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:28.239782 (+ 81858us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
W20260927 11:15:28.240163 15388 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38695 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38695: Client connection negotiation failed: client connection to 127.0.98.62:38695: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 11:15:28.240492 15390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:28.240382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:56023)
0927 11:15:28.240431 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:28.240473 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 11:15:28.240871 15390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:28.240800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:47463)
0927 11:15:28.240836 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:28.240857 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 11:15:28.240984 15388 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 11:15:28.244140 15392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:28.244323 15392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:28.244343 15392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:28.245397 15392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:28.245428 15392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:28.245455 15392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:15:28.246843 15392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34873
--tserver_master_addrs=127.0.98.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:28.247017 15392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:28.247175 15392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:28.251667 15398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:28.251703 15399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:28.251756 15401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:28.251802 15392 server_base.cc:1034] running on GCE node
I20260927 11:15:28.252095 15392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:28.252324 15392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:28.253472 15392 hybrid_clock.cc:648] HybridClock initialized: now 1790507728253464 us; error 34 us; skew 500 ppm
Sep 27 11:15:28 dist-test-slave-9hq9 krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:28.255777 15392 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:15:28.256197 15392 webserver.cc:460] Webserver started at http://127.0.98.2:34873/ using document root <none> and password file <none>
I20260927 11:15:28.256417 15392 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:28.256480 15392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:28.257731 15392 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:15:28.258315 15408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:28.258484 15392 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:15:28.258569 15392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "e04011c612004fabbdd89cefc40e4083"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "2a19acea7d8b405a1890b79b94084975"
server_key_iv: "4118f28a9f6b3fa575952cd5704bfd69"
server_key_version: "encryptionkey@0"
I20260927 11:15:28.258870 15392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:28.272753 15392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:28.274421 15392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:28.274566 15392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:28.274760 15392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:28.275023 15392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:15:28.275071 15392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:28.275110 15392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:15:28.275138 15392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:28.280560 15392 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40749
I20260927 11:15:28.280602 15521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40749 every 8 connection(s)
I20260927 11:15:28.280907 15392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:15:28.281934 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:28.281568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:43633)
0927 11:15:28.281695 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:28.281769 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
W20260927 11:15:28.282308 15522 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38695 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38695: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
I20260927 11:15:28.282552 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:28.282471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:48143)
0927 11:15:28.282509 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:28.282534 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 11:15:28.282765 15524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:28.282718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:37169)
0927 11:15:28.282737 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:28.282754 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260927 11:15:28.282850 15522 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 11:15:28.288307 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 15392
I20260927 11:15:28.288565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45691
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38695
--webserver_port=46549
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:28.365667 15525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:28.365870 15525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:28.365907 15525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:28.367028 15525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:28.367093 15525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:28.367136 15525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:15:28.368517 15525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46549
--tserver_master_addrs=127.0.98.62:38695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:28.368768 15525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:28.368966 15525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:28.373620 15533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:28.373627 15531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:28.373656 15525 server_base.cc:1034] running on GCE node
W20260927 11:15:28.373627 15530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:28.374043 15525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:28.374305 15525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:28.375473 15525 hybrid_clock.cc:648] HybridClock initialized: now 1790507728375462 us; error 38 us; skew 500 ppm
Sep 27 11:15:28 dist-test-slave-9hq9 krb5kdc[14090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:28.377622 15525 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:15:28.378003 15525 webserver.cc:460] Webserver started at http://127.0.98.3:46549/ using document root <none> and password file <none>
I20260927 11:15:28.378253 15525 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:28.378317 15525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:28.379521 15525 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:15:28.380028 15540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:28.380183 15525 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:15:28.380270 15525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "b31be1cacc454e18a5cc67d33cab6c61"
format_stamp: "Formatted at 2026-09-27 11:15:26 on dist-test-slave-9hq9"
server_key: "3588a434c9b3fad7e5e8835ada0e327b"
server_key_iv: "d0b3b4711cc7ad0b94360f2724dfdd6d"
server_key_version: "encryptionkey@0"
I20260927 11:15:28.380563 15525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:28.398269 15525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:28.399819 15525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:28.399956 15525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:28.400146 15525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:28.400403 15525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:15:28.400449 15525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:28.400493 15525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:15:28.400520 15525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:28.405922 15525 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45691
I20260927 11:15:28.405987 15653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45691 every 8 connection(s)
I20260927 11:15:28.406307 15525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:15:28.407228 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:28.406841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:55899)
0927 11:15:28.406972 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:28.407063 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
W20260927 11:15:28.407564 15654 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38695 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38695: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
I20260927 11:15:28.407810 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:28.407730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:55899)
0927 11:15:28.407771 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:28.407795 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 11:15:28.408047 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:28.407980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:52757)
0927 11:15:28.407999 (+ 19us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:28.408014 (+ 15us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260927 11:15:28.408146 15654 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 11:15:28.413120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 15525
I20260927 11:15:29.241488 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:29.241288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:35441)
0927 11:15:29.241392 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:29.241427 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260927 11:15:29.283298 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:29.283105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:47365)
0927 11:15:29.283207 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:29.283253 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":51,"threads_started":1}
I20260927 11:15:29.408598 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:29.408426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:59421)
0927 11:15:29.408528 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:29.408559 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260927 11:15:30.000332 15661 raft_consensus.cc:482] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:15:30.000459 15661 raft_consensus.cc:504] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } }
I20260927 11:15:30.000819 15661 leader_election.cc:288] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b31be1cacc454e18a5cc67d33cab6c61 (127.0.98.3:45691), e04011c612004fabbdd89cefc40e4083 (127.0.98.2:40749)
Sep 27 11:15:30 dist-test-slave-9hq9 krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 11:15:30 dist-test-slave-9hq9 krb5kdc[14090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 11:15:30.010888 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:30.000989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40749 (local address 127.0.98.1:51743)
0927 11:15:30.001108 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:30.001119 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:30.001169 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:30.001949 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:30.001950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:30.002322 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:30.002505 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:30.002511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:30.003563 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:30.003564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:30.004296 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:30.004298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:30.004862 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:30.004864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:30.004970 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:30.005585 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:30.005589 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:30.007023 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:30.009264 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:30.009271 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:30.009272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:30.009696 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:30.009967 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:30.009967 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:30.009968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:30.010036 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:30.010433 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:30.010434 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:30.010527 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 11:15:30.010619 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 11:15:30.010791 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":50,"thread_start_us":50,"threads_started":1}
I20260927 11:15:30.010910 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:30.001054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40213 (local address 127.0.98.3:45691)
0927 11:15:30.001265 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:30.001273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:30.001291 (+ 18us) server_negotiation.cc:407] Connection header received
0927 11:15:30.001704 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:30.001706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:30.001750 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:30.001854 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:30.002634 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:30.003447 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:30.004398 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:30.004791 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:30.004914 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:30.007539 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:30.007542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:30.007542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:30.007551 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:30.009338 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:30.009764 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:30.009766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:30.009767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:30.009860 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:30.010134 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:30.010135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:30.010135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:30.010344 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:30.010426 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:30.010686 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 11:15:30.010799 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":82,"threads_started":1}
I20260927 11:15:30.010926 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:30.001105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51743 (local address 127.0.98.2:40749)
0927 11:15:30.001261 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:30.001272 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:30.001291 (+ 19us) server_negotiation.cc:407] Connection header received
0927 11:15:30.001703 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:30.001705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:30.001744 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:30.001884 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:30.002611 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:30.003449 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:30.004404 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:30.004785 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:30.004885 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:30.007133 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:30.007137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:30.007137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:30.007148 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:30.009187 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:30.009802 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:30.009804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:30.009805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:30.009895 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:30.010128 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:30.010130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:30.010130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:30.010351 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:30.010437 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:30.010684 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 11:15:30.010819 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":53,"threads_started":1}
I20260927 11:15:30.010993 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:30.000956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45691 (local address 127.0.98.1:40213)
0927 11:15:30.001068 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:30.001079 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:30.001139 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:30.001917 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:30.001920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:30.002319 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:30.002505 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:30.002511 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:30.003536 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:30.003538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:30.004296 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:30.004297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:30.004886 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:30.004887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:30.004972 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:30.005519 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:30.005526 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:30.007463 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:30.009412 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:30.009413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:30.009413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:30.009657 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:30.009926 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:30.009926 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:30.009927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:30.010024 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:30.010411 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:30.010417 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:30.010527 (+ 110us) client_negotiation.cc:769] Sending connection context
0927 11:15:30.010619 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 11:15:30.010871 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":137,"thread_start_us":43,"threads_started":1}
W20260927 11:15:30.012526 15276 leader_election.cc:341] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e04011c612004fabbdd89cefc40e4083 (127.0.98.2:40749): Not found: Tablet not found: a953c78dfdeb4adaa22638f94e016ba9
W20260927 11:15:30.012614 15275 leader_election.cc:341] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b31be1cacc454e18a5cc67d33cab6c61 (127.0.98.3:45691): Not found: Tablet not found: a953c78dfdeb4adaa22638f94e016ba9
I20260927 11:15:30.012665 15275 leader_election.cc:302] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 220afa49dff44ded9274015f228645f8; no voters: b31be1cacc454e18a5cc67d33cab6c61, e04011c612004fabbdd89cefc40e4083
I20260927 11:15:30.012751 15661 raft_consensus.cc:2726] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 11:15:30.241972 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:30.241856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:58385)
0927 11:15:30.241907 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:30.241937 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 11:15:30.283885 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:30.283646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:34559)
0927 11:15:30.283786 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:30.283848 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260927 11:15:30.409186 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:30.408979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:53411)
0927 11:15:30.409106 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:30.409147 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260927 11:15:31.242539 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:31.242357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:52767)
0927 11:15:31.242466 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:31.242497 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260927 11:15:31.284487 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:31.284312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:59761)
0927 11:15:31.284431 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:31.284454 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260927 11:15:31.409814 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:31.409593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:36873)
0927 11:15:31.409718 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:31.409769 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260927 11:15:31.711689 15671 raft_consensus.cc:482] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:15:31.711843 15671 raft_consensus.cc:504] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } }
I20260927 11:15:31.712047 15671 leader_election.cc:288] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b31be1cacc454e18a5cc67d33cab6c61 (127.0.98.3:45691), e04011c612004fabbdd89cefc40e4083 (127.0.98.2:40749)
W20260927 11:15:31.712416 15275 leader_election.cc:341] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b31be1cacc454e18a5cc67d33cab6c61 (127.0.98.3:45691): Not found: Tablet not found: a953c78dfdeb4adaa22638f94e016ba9
W20260927 11:15:31.712486 15276 leader_election.cc:341] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e04011c612004fabbdd89cefc40e4083 (127.0.98.2:40749): Not found: Tablet not found: a953c78dfdeb4adaa22638f94e016ba9
I20260927 11:15:31.712535 15275 leader_election.cc:302] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 220afa49dff44ded9274015f228645f8; no voters: b31be1cacc454e18a5cc67d33cab6c61, e04011c612004fabbdd89cefc40e4083
I20260927 11:15:31.712601 15671 raft_consensus.cc:2726] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 11:15:32.243141 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:32.242883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:52623)
0927 11:15:32.243037 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:32.243068 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260927 11:15:32.284953 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:32.284811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:45235)
0927 11:15:32.284897 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:32.284923 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":45,"threads_started":1}
I20260927 11:15:32.410423 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:32.410221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:37725)
0927 11:15:32.410340 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:32.410372 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260927 11:15:33.243764 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:33.243560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:32793)
0927 11:15:33.243691 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:33.243720 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260927 11:15:33.285501 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:33.285304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:57917)
0927 11:15:33.285413 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:33.285467 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
I20260927 11:15:33.410970 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:33.410773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:46131)
0927 11:15:33.410906 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:33.410935 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260927 11:15:34.244359 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:34.244150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:56615)
0927 11:15:34.244255 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:34.244284 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260927 11:15:34.286018 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:34.285858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:59923)
0927 11:15:34.285960 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:34.285988 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
I20260927 11:15:34.411571 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:34.411406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:40493)
0927 11:15:34.411507 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:34.411535 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":55,"threads_started":1}
I20260927 11:15:34.605846 15681 raft_consensus.cc:482] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:15:34.605928 15681 raft_consensus.cc:504] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } }
I20260927 11:15:34.606138 15681 leader_election.cc:288] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b31be1cacc454e18a5cc67d33cab6c61 (127.0.98.3:45691), e04011c612004fabbdd89cefc40e4083 (127.0.98.2:40749)
W20260927 11:15:34.606467 15276 leader_election.cc:341] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e04011c612004fabbdd89cefc40e4083 (127.0.98.2:40749): Not found: Tablet not found: a953c78dfdeb4adaa22638f94e016ba9
W20260927 11:15:34.606614 15275 leader_election.cc:341] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b31be1cacc454e18a5cc67d33cab6c61 (127.0.98.3:45691): Not found: Tablet not found: a953c78dfdeb4adaa22638f94e016ba9
I20260927 11:15:34.606678 15275 leader_election.cc:302] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 220afa49dff44ded9274015f228645f8; no voters: b31be1cacc454e18a5cc67d33cab6c61, e04011c612004fabbdd89cefc40e4083
I20260927 11:15:34.606742 15681 raft_consensus.cc:2726] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 11:15:35.244896 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:35.244709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:59625)
0927 11:15:35.244816 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:35.244844 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260927 11:15:35.286635 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:35.286465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:54581)
0927 11:15:35.286571 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:35.286601 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260927 11:15:35.412099 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:35.411916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:48277)
0927 11:15:35.412032 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:35.412064 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":52,"threads_started":1}
I20260927 11:15:36.245455 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:36.245273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:53797)
0927 11:15:36.245383 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:36.245412 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260927 11:15:36.287223 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:36.287038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:48583)
0927 11:15:36.287155 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:36.287190 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260927 11:15:36.412683 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:36.412502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:45739)
0927 11:15:36.412613 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:36.412646 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
I20260927 11:15:37.246008 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:37.245821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:44335)
0927 11:15:37.245934 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:37.245964 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260927 11:15:37.287801 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:37.287617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:60301)
0927 11:15:37.287732 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:37.287768 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
I20260927 11:15:37.413298 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:37.413106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:48045)
0927 11:15:37.413220 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:37.413258 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260927 11:15:38.246685 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:38.246425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:48937)
0927 11:15:38.246586 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:38.246629 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260927 11:15:38.288457 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:38.288274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:51531)
0927 11:15:38.288396 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:38.288428 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":52,"threads_started":1}
I20260927 11:15:38.413939 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:38.413700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:53623)
0927 11:15:38.413833 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:38.413890 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260927 11:15:38.432432 15694 raft_consensus.cc:482] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:15:38.432513 15694 raft_consensus.cc:504] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31be1cacc454e18a5cc67d33cab6c61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45691 } } peers { permanent_uuid: "e04011c612004fabbdd89cefc40e4083" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40749 } } peers { permanent_uuid: "220afa49dff44ded9274015f228645f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43815 } }
I20260927 11:15:38.432766 15694 leader_election.cc:288] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b31be1cacc454e18a5cc67d33cab6c61 (127.0.98.3:45691), e04011c612004fabbdd89cefc40e4083 (127.0.98.2:40749)
W20260927 11:15:38.433076 15276 leader_election.cc:341] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e04011c612004fabbdd89cefc40e4083 (127.0.98.2:40749): Not found: Tablet not found: a953c78dfdeb4adaa22638f94e016ba9
W20260927 11:15:38.433161 15275 leader_election.cc:341] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b31be1cacc454e18a5cc67d33cab6c61 (127.0.98.3:45691): Not found: Tablet not found: a953c78dfdeb4adaa22638f94e016ba9
I20260927 11:15:38.433231 15275 leader_election.cc:302] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 220afa49dff44ded9274015f228645f8; no voters: b31be1cacc454e18a5cc67d33cab6c61, e04011c612004fabbdd89cefc40e4083
I20260927 11:15:38.433321 15694 raft_consensus.cc:2726] T a953c78dfdeb4adaa22638f94e016ba9 P 220afa49dff44ded9274015f228645f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 11:15:39.247236 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:39.247045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:44097)
0927 11:15:39.247159 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:39.247191 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260927 11:15:39.289197 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:39.289004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:48445)
0927 11:15:39.289128 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:39.289162 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260927 11:15:39.414515 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:39.414348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:53615)
0927 11:15:39.414449 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:39.414477 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":53,"threads_started":1}
I20260927 11:15:40.247793 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:40.247619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:36169)
0927 11:15:40.247722 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:40.247751 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":54,"threads_started":1}
I20260927 11:15:40.289754 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:40.289581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:34429)
0927 11:15:40.289690 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:40.289723 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":52,"threads_started":1}
I20260927 11:15:40.415068 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:40.414874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:54873)
0927 11:15:40.415004 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:40.415034 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260927 11:15:41.248494 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:41.248243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:38699)
0927 11:15:41.248395 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:41.248438 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
I20260927 11:15:41.290436 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:41.290268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:43063)
0927 11:15:41.290373 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:41.290405 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":52,"threads_started":1}
I20260927 11:15:41.415616 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:41.415421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:57551)
0927 11:15:41.415551 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:41.415581 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":53,"threads_started":1}
I20260927 11:15:42.249136 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:42.248935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:54401)
0927 11:15:42.249050 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:42.249094 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":52,"threads_started":1}
I20260927 11:15:42.291011 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:42.290839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:44731)
0927 11:15:42.290948 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:42.290980 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":51,"threads_started":1}
I20260927 11:15:42.416273 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:42.416090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:57805)
0927 11:15:42.416208 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:42.416238 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260927 11:15:43.250001 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:43.249719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.1:52335)
0927 11:15:43.249924 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:43.249958 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":68,"threads_started":1}
I20260927 11:15:43.291697 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:43.291532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.2:46331)
0927 11:15:43.291639 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:43.291665 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
I20260927 11:15:43.414860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 15229
I20260927 11:15:43.416889 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:43.416719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38695 (local address 127.0.98.3:40869)
0927 11:15:43.416826 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:43.416856 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
I20260927 11:15:43.419708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 15392
I20260927 11:15:43.423755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 15525
2026-09-27T11:15:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18460 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:15:43 dist-test-slave-9hq9 krb5kdc[15712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:15:43 dist-test-slave-9hq9 krb5kdc[15712](info): set up 2 sockets
Sep 27 11:15:43 dist-test-slave-9hq9 krb5kdc[15712](info): commencing operation
krb5kdc: starting...
W20260927 11:15:44.453905 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:15:44 dist-test-slave-9hq9 krb5kdc[15712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:15:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:15:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:15:44.566848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36613
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:44.642779 15725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:44.643003 15725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:44.643045 15725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:44.644161 15725 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 11:15:44.644220 15725 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 11:15:44.644243 15725 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 11:15:44.644272 15725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:15:44.644301 15725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:44.644330 15725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:44.644358 15725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:15:44.644387 15725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:15:44.645758 15725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:44.646000 15725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:44.646231 15725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:44.650974 15730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:44.651072 15733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:44.651023 15731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:44.650985 15725 server_base.cc:1034] running on GCE node
I20260927 11:15:44.651346 15725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:44.651659 15725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:44.652812 15725 hybrid_clock.cc:648] HybridClock initialized: now 1790507744652790 us; error 39 us; skew 500 ppm
Sep 27 11:15:44 dist-test-slave-9hq9 krb5kdc[15712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:44.655210 15725 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:15:44.655551 15725 webserver.cc:460] Webserver started at http://127.0.98.62:45435/ using document root <none> and password file <none>
I20260927 11:15:44.655715 15725 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:44.655753 15725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:44.655833 15725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:15:44.656701 15725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "f1a6b000a5af497db84cbd05c73d9437"
format_stamp: "Formatted at 2026-09-27 11:15:44 on dist-test-slave-9hq9"
server_key: "5688aaac04e53e6a6d5d8215b1ec06d9"
server_key_iv: "4979776320e4a40f6a55056e469ac6c5"
server_key_version: "encryptionkey@0"
I20260927 11:15:44.656970 15725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1a6b000a5af497db84cbd05c73d9437"
format_stamp: "Formatted at 2026-09-27 11:15:44 on dist-test-slave-9hq9"
server_key: "5688aaac04e53e6a6d5d8215b1ec06d9"
server_key_iv: "4979776320e4a40f6a55056e469ac6c5"
server_key_version: "encryptionkey@0"
I20260927 11:15:44.658179 15725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:15:44.658802 15740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:44.658962 15725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:15:44.659055 15725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "f1a6b000a5af497db84cbd05c73d9437"
format_stamp: "Formatted at 2026-09-27 11:15:44 on dist-test-slave-9hq9"
server_key: "5688aaac04e53e6a6d5d8215b1ec06d9"
server_key_iv: "4979776320e4a40f6a55056e469ac6c5"
server_key_version: "encryptionkey@0"
I20260927 11:15:44.659148 15725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:44.670254 15725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:44.671955 15725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:44.672065 15725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:44.675714 15725 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36613
I20260927 11:15:44.675741 15792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36613 every 8 connection(s)
I20260927 11:15:44.676045 15725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:15:44.676844 15793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:44.678881 15793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Bootstrap starting.
I20260927 11:15:44.679405 15793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:44.679684 15793 log.cc:824] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:44.680284 15793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: No bootstrap required, opened a new log
I20260927 11:15:44.680785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 15725
I20260927 11:15:44.680855 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:15:44.681010 392 external_mini_cluster.cc:1428] Setting key 7ca280862ecf14404777a83f9bc62cf3
I20260927 11:15:44.681949 15793 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 11:15:44.682127 15793 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:44.682168 15793 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a6b000a5af497db84cbd05c73d9437, State: Initialized, Role: FOLLOWER
I20260927 11:15:44.682324 15793 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 11:15:44.682417 15793 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:15:44.682454 15793 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:15:44.682508 15793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:44.683259 15793 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 11:15:44.683373 15793 leader_election.cc:302] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1a6b000a5af497db84cbd05c73d9437; no voters:
I20260927 11:15:44.683543 15793 leader_election.cc:288] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:15:44.683578 15798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:15:44.683676 15798 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 1 LEADER]: Becoming Leader. State: Replica: f1a6b000a5af497db84cbd05c73d9437, State: Running, Role: LEADER
I20260927 11:15:44.683784 15793 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:15:44.683808 15798 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 11:15:44.684182 15798 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1a6b000a5af497db84cbd05c73d9437. Latest consensus state: current_term: 1 leader_uuid: "f1a6b000a5af497db84cbd05c73d9437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } } }
I20260927 11:15:44.684192 15799 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1a6b000a5af497db84cbd05c73d9437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } } }
I20260927 11:15:44.684317 15798 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:44.684404 15799 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [sys.catalog]: This master's current role is: LEADER
I20260927 11:15:44.684580 15807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:15:44.685062 15807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:15:44.686751 15807 catalog_manager.cc:1347] Generated new cluster ID: e790921a47884b6fb0b99b84418e1e5f
I20260927 11:15:44.686800 15807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:15:44 dist-test-slave-9hq9 krb5kdc[15712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:44.690178 15807 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:15:44.690819 15807 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:15:44.692150 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:44.681464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.0.98.62:36613)
0927 11:15:44.681652 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:44.681661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:44.681693 (+ 32us) server_negotiation.cc:407] Connection header received
0927 11:15:44.681940 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:44.681945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:44.681984 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:44.682174 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:44.682852 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:44.684005 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:44.685150 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:44.685834 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:44.685968 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:44.688613 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:44.688616 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:44.688616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:44.688626 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:44.690603 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:44.691000 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:44.691002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:44.691003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:44.691104 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:44.691331 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:44.691331 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:44.691332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:44.691598 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:44.691727 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:44.691851 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 11:15:44.691978 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260927 11:15:44.698493 15807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Generated new TSK 0
I20260927 11:15:44.698632 15807 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:15:44.725275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36613
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:44.801188 15819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:44.801378 15819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:44.801399 15819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:44.802376 15819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:44.802407 15819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:44.802429 15819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:15:44.803704 15819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:44.803856 15819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:44.804008 15819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:44.808445 15824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:44.808431 15825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:44.808531 15819 server_base.cc:1034] running on GCE node
W20260927 11:15:44.808429 15827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:44.808837 15819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:44.809026 15819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:44.810122 15819 hybrid_clock.cc:648] HybridClock initialized: now 1790507744810109 us; error 27 us; skew 500 ppm
Sep 27 11:15:44 dist-test-slave-9hq9 krb5kdc[15712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:44.812393 15819 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:15:44.812786 15819 webserver.cc:460] Webserver started at http://127.0.98.1:43733/ using document root <none> and password file <none>
I20260927 11:15:44.812973 15819 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:44.813011 15819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:44.813099 15819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:15:44.813962 15819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "a9d37f7a1207453b94b44ea2849bbabc"
format_stamp: "Formatted at 2026-09-27 11:15:44 on dist-test-slave-9hq9"
server_key: "1440bebb20a1de3b6fe6727c21f2fa2f"
server_key_iv: "b1aedf4c1c453bfd29725f0a00a5e458"
server_key_version: "encryptionkey@0"
I20260927 11:15:44.814324 15819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a9d37f7a1207453b94b44ea2849bbabc"
format_stamp: "Formatted at 2026-09-27 11:15:44 on dist-test-slave-9hq9"
server_key: "1440bebb20a1de3b6fe6727c21f2fa2f"
server_key_iv: "b1aedf4c1c453bfd29725f0a00a5e458"
server_key_version: "encryptionkey@0"
I20260927 11:15:44.815460 15819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:15:44.816066 15834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:44.816246 15819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:15:44.816339 15819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "a9d37f7a1207453b94b44ea2849bbabc"
format_stamp: "Formatted at 2026-09-27 11:15:44 on dist-test-slave-9hq9"
server_key: "1440bebb20a1de3b6fe6727c21f2fa2f"
server_key_iv: "b1aedf4c1c453bfd29725f0a00a5e458"
server_key_version: "encryptionkey@0"
I20260927 11:15:44.816411 15819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:44.830365 15819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:44.831853 15819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:44.831995 15819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:44.832222 15819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:44.832485 15819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:15:44.832536 15819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:44.832573 15819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:15:44.832603 15819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:44.837896 15819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40657
I20260927 11:15:44.837932 15947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40657 every 8 connection(s)
I20260927 11:15:44.838248 15819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:15:44.839049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 15819
I20260927 11:15:44.839164 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:15:44.839323 392 external_mini_cluster.cc:1428] Setting key 3e6a94910a8bf41145cc58560bd8d005
Sep 27 11:15:44 dist-test-slave-9hq9 krb5kdc[15712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 11:15:44.847995 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:44.838901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42761 (local address 127.0.98.62:36613)
0927 11:15:44.838963 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:44.838965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:44.839172 (+ 207us) server_negotiation.cc:407] Connection header received
0927 11:15:44.839710 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:44.839711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:44.839726 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:44.839776 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:44.840743 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:44.841464 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:44.842486 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:44.842839 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:44.842910 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:44.845249 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:44.845250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:44.845251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:44.845256 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:44.846601 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:44.847091 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:44.847091 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:44.847092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:44.847140 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:44.847410 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:44.847411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:44.847411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:44.847502 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:44.847550 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:44.847867 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 11:15:44.847934 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:15:44.848253 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:44.838878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:42761)
0927 11:15:44.839030 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:44.839066 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:44.839604 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:44.839833 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:44.839837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:44.840448 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:44.840630 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:44.840634 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:44.841556 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:44.841556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:44.842409 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:44.842412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:44.842918 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:44.842919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:44.843006 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:44.843482 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:44.843490 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:44.845160 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:44.846687 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:44.846695 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:44.846697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:44.846998 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:44.847208 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:44.847209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:44.847209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:44.847341 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:44.847562 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:44.847569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:44.847728 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 11:15:44.847850 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 11:15:44.848011 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260927 11:15:44.848716 15948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36613
I20260927 11:15:44.848799 15948 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:44.848920 15948 heartbeater.cc:511] Master 127.0.98.62:36613 requested a full tablet report, sending...
I20260927 11:15:44.849459 15757 ts_manager.cc:194] Registered new tserver with Master: a9d37f7a1207453b94b44ea2849bbabc (127.0.98.1:40657)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 11:15:44.850201 15757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:15:44.865345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36613
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:44.942945 15953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:44.943156 15953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:44.943194 15953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:44.944365 15953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:44.944427 15953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:44.944468 15953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:15:44.945876 15953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:44.946162 15953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:44.946376 15953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:44.951074 15959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:44.951107 15961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:44.951077 15958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:44.952154 15953 server_base.cc:1034] running on GCE node
I20260927 11:15:44.952369 15953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:44.952600 15953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:44.953755 15953 hybrid_clock.cc:648] HybridClock initialized: now 1790507744953740 us; error 27 us; skew 500 ppm
Sep 27 11:15:44 dist-test-slave-9hq9 krb5kdc[15712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:44.956236 15953 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:15:44.956657 15953 webserver.cc:460] Webserver started at http://127.0.98.2:46825/ using document root <none> and password file <none>
I20260927 11:15:44.956822 15953 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:44.956861 15953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:44.956952 15953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:15:44.957870 15953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "a7beba48c03a4d7fae7e832fa1384bd6"
format_stamp: "Formatted at 2026-09-27 11:15:44 on dist-test-slave-9hq9"
server_key: "fe46c923fbb4d83739d9a8ab1541ed6c"
server_key_iv: "5d094f06aa213a5f054fe2f5a1899068"
server_key_version: "encryptionkey@0"
I20260927 11:15:44.958233 15953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a7beba48c03a4d7fae7e832fa1384bd6"
format_stamp: "Formatted at 2026-09-27 11:15:44 on dist-test-slave-9hq9"
server_key: "fe46c923fbb4d83739d9a8ab1541ed6c"
server_key_iv: "5d094f06aa213a5f054fe2f5a1899068"
server_key_version: "encryptionkey@0"
I20260927 11:15:44.959518 15953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:15:44.960117 15968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:44.960285 15953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:15:44.960352 15953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "a7beba48c03a4d7fae7e832fa1384bd6"
format_stamp: "Formatted at 2026-09-27 11:15:44 on dist-test-slave-9hq9"
server_key: "fe46c923fbb4d83739d9a8ab1541ed6c"
server_key_iv: "5d094f06aa213a5f054fe2f5a1899068"
server_key_version: "encryptionkey@0"
I20260927 11:15:44.960398 15953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:44.971323 15953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:44.973453 15953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:44.973556 15953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:44.973711 15953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:44.973930 15953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:15:44.973958 15953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:44.973978 15953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:15:44.973990 15953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:44.979684 15953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40795
I20260927 11:15:44.979720 16081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40795 every 8 connection(s)
I20260927 11:15:44.980015 15953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:15:44 dist-test-slave-9hq9 krb5kdc[15712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:44.988222 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:44.980628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34825 (local address 127.0.98.62:36613)
0927 11:15:44.980699 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:44.980701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:44.980855 (+ 154us) server_negotiation.cc:407] Connection header received
0927 11:15:44.981260 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:44.981261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:44.981270 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:44.981339 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:44.982213 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:44.982679 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:44.983630 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:44.983850 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:44.983900 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:44.985987 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:44.985988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:44.985989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:44.985995 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:44.987163 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:44.987501 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:44.987502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:44.987502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:44.987548 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:44.987746 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:44.987746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:44.987746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:44.987840 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:44.987887 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:44.988106 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 11:15:44.988162 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 11:15:44.988390 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:44.980587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:34825)
0927 11:15:44.980752 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:44.980782 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:44.981188 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:44.981391 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:44.981394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:44.981905 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:44.982134 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:44.982139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:44.982782 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:44.982782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:44.983567 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:44.983569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:44.983927 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:44.983928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:44.984021 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:44.984508 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:44.984515 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:44.985897 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:44.987230 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:44.987235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:44.987236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:44.987438 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:44.987605 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:44.987605 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:44.987606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:44.987693 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:44.987892 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:44.987896 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:44.988000 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 11:15:44.988075 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 11:15:44.988180 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260927 11:15:44.988930 16082 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36613
I20260927 11:15:44.989013 16082 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:44.989192 16082 heartbeater.cc:511] Master 127.0.98.62:36613 requested a full tablet report, sending...
I20260927 11:15:44.989537 15757 ts_manager.cc:194] Registered new tserver with Master: a7beba48c03a4d7fae7e832fa1384bd6 (127.0.98.2:40795)
I20260927 11:15:44.989811 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 15953
I20260927 11:15:44.989913 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:15:44.990042 15757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34825
I20260927 11:15:44.990103 392 external_mini_cluster.cc:1428] Setting key d46ce309d19ef21d13f382813f6bc746
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:15:45.012531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36613
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:15:45.089032 16087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:15:45.089227 16087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:15:45.089262 16087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:15:45.090382 16087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:15:45.090445 16087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:15:45.090487 16087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:15:45.091902 16087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:15:45.092136 16087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:15:45.092340 16087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:15:45.096894 16095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:45.096902 16093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:15:45.096926 16092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:15:45.097184 16087 server_base.cc:1034] running on GCE node
I20260927 11:15:45.097373 16087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:15:45.097600 16087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:15:45.098729 16087 hybrid_clock.cc:648] HybridClock initialized: now 1790507745098717 us; error 27 us; skew 500 ppm
Sep 27 11:15:45 dist-test-slave-9hq9 krb5kdc[15712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:15:45.101029 16087 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:15:45.101456 16087 webserver.cc:460] Webserver started at http://127.0.98.3:42989/ using document root <none> and password file <none>
I20260927 11:15:45.101675 16087 fs_manager.cc:359] Metadata directory not provided
I20260927 11:15:45.101737 16087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:15:45.101879 16087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:15:45.102877 16087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "5c3c81a6cd1c49f6a05a952c5f81207f"
format_stamp: "Formatted at 2026-09-27 11:15:45 on dist-test-slave-9hq9"
server_key: "a2126f507d59181b2c150a2052a99e04"
server_key_iv: "9c82276b27709e9741832ba456c5a4af"
server_key_version: "encryptionkey@0"
I20260927 11:15:45.103225 16087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5c3c81a6cd1c49f6a05a952c5f81207f"
format_stamp: "Formatted at 2026-09-27 11:15:45 on dist-test-slave-9hq9"
server_key: "a2126f507d59181b2c150a2052a99e04"
server_key_iv: "9c82276b27709e9741832ba456c5a4af"
server_key_version: "encryptionkey@0"
I20260927 11:15:45.104439 16087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:15:45.105108 16102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:45.105273 16087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:15:45.105363 16087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "5c3c81a6cd1c49f6a05a952c5f81207f"
format_stamp: "Formatted at 2026-09-27 11:15:45 on dist-test-slave-9hq9"
server_key: "a2126f507d59181b2c150a2052a99e04"
server_key_iv: "9c82276b27709e9741832ba456c5a4af"
server_key_version: "encryptionkey@0"
I20260927 11:15:45.105450 16087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:15:45.132718 16087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:15:45.134970 16087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:15:45.135128 16087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:15:45.135339 16087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:15:45.135634 16087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:15:45.135685 16087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:45.135718 16087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:15:45.135746 16087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:15:45.141275 16087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40583
I20260927 11:15:45.141326 16215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40583 every 8 connection(s)
I20260927 11:15:45.141604 16087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:15:45.146734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 16087
I20260927 11:15:45.146842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
Sep 27 11:15:45 dist-test-slave-9hq9 krb5kdc[15712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:15:45.147014 392 external_mini_cluster.cc:1428] Setting key 8838457a57733231063f200a7883b42e
I20260927 11:15:45.150020 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:45.142269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33805 (local address 127.0.98.62:36613)
0927 11:15:45.142340 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:45.142342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:45.142534 (+ 192us) server_negotiation.cc:407] Connection header received
0927 11:15:45.142981 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:45.142982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:45.142992 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:45.143069 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:45.143885 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.144366 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.145273 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.145454 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.145503 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:45.147522 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:45.147523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:45.147523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:45.147531 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:45.148987 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:45.149331 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:45.149331 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:45.149332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:45.149380 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:45.149541 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:45.149541 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:45.149542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:45.149628 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:45.149691 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:45.149888 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 11:15:45.149959 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 11:15:45.150235 16218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:45.142255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:33805)
0927 11:15:45.142415 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:45.142452 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:45.142901 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:45.143115 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:45.143118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:45.143635 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:15:45.143804 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:45.143808 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.144441 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.144442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:45.145199 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:45.145201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.145520 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.145521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:45.145618 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:45.146013 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:15:45.146021 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:15:45.147446 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:15:45.149054 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:45.149071 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:45.149073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:45.149269 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:45.149426 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:15:45.149427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:15:45.149427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:15:45.149501 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:15:45.149697 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:15:45.149701 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:15:45.149801 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 11:15:45.149879 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 11:15:45.149986 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260927 11:15:45.150694 16216 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36613
I20260927 11:15:45.150766 16216 heartbeater.cc:464] Registering TS with master...
I20260927 11:15:45.150949 16216 heartbeater.cc:511] Master 127.0.98.62:36613 requested a full tablet report, sending...
I20260927 11:15:45.151306 15757 ts_manager.cc:194] Registered new tserver with Master: 5c3c81a6cd1c49f6a05a952c5f81207f (127.0.98.3:40583)
I20260927 11:15:45.151680 15757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33805
I20260927 11:15:45.158979 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:15:45.166234 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:45.160239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53014 (local address 127.0.98.62:36613)
0927 11:15:45.160293 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:45.160294 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:45.160425 (+ 131us) server_negotiation.cc:407] Connection header received
0927 11:15:45.160459 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:45.160460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:45.160468 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:45.160511 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:15:45.161017 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.161494 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.162255 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.162487 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.162536 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:45.163666 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:15:45.163667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:15:45.163667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:15:45.163673 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:15:45.165153 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:45.165567 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:45.165568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:45.165568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:45.165644 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:15:45.165801 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:15:45.165801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:15:45.165802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:15:45.165913 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:15:45.165980 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:45.166113 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 11:15:45.166180 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:15:45.167593 15757 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:15:45.169066 15757 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:15:45.175235 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:45.171276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.0.98.3:40583)
0927 11:15:45.171760 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:45.171763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:45.171772 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:15:45.171804 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:45.171806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:45.171820 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:45.171889 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:45.172013 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.173102 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.174376 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.174837 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.174934 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:45.174989 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:45.175116 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 11:15:45.175154 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":80,"threads_started":1}
I20260927 11:15:45.175253 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:45.171120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40583 (local address 127.0.0.1:40228)
0927 11:15:45.171392 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:45.171435 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:45.171498 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:45.171887 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:45.171888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:45.171890 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:45.171935 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:45.171937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.173201 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.173202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:45.174292 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:45.174294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.174936 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.174937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:45.175016 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:45.175027 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 11:15:45.175097 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 11:15:45.175149 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":44,"thread_start_us":65,"threads_started":1}
I20260927 11:15:45.175426 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:45.171376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.0.98.2:40795)
0927 11:15:45.171606 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:45.171610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:45.171622 (+ 12us) server_negotiation.cc:407] Connection header received
0927 11:15:45.171661 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:45.171664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:45.171680 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:45.171763 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:45.171937 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.172738 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.174169 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.174865 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.174948 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:45.175000 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:45.175293 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 11:15:45.175336 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260927 11:15:45.175453 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:45.171207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40795 (local address 127.0.0.1:51576)
0927 11:15:45.171542 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:45.171552 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:45.171592 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:45.171781 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:45.171784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:45.171786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:45.171846 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:45.171848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.172885 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.172886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:45.174086 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:45.174089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.175035 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.175036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:45.175095 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:45.175099 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:15:45.175147 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 11:15:45.175193 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":2,"spinlock_wait_cycles":896,"thread_start_us":97,"threads_started":1}
I20260927 11:15:45.175556 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:45.171207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40657 (local address 127.0.0.1:56922)
0927 11:15:45.171535 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:45.171546 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:45.171634 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:45.172091 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:45.172092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:45.172094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:45.172136 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:45.172137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.173375 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.173377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:45.174386 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:45.174389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.175309 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.175310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:45.175362 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:45.175364 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:15:45.175425 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 11:15:45.175466 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260927 11:15:45.175782 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:45.171360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.0.98.1:40657)
0927 11:15:45.171520 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:45.171525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:45.171605 (+ 80us) server_negotiation.cc:407] Connection header received
0927 11:15:45.171716 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:45.171718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:45.171751 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:45.172014 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:45.172199 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.173280 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.174484 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.175173 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.175229 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:45.175277 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:45.175691 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 11:15:45.175725 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260927 11:15:45.176388 16016 tablet_service.cc:1467] Processing CreateTablet for tablet 86ebb722050a4538b103418c6beb9347 (DEFAULT_TABLE table=test-table [id=0c0f5ecb769541c9b641e0625ea244d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:15:45.176484 15882 tablet_service.cc:1467] Processing CreateTablet for tablet 86ebb722050a4538b103418c6beb9347 (DEFAULT_TABLE table=test-table [id=0c0f5ecb769541c9b641e0625ea244d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:15:45.176731 16016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ebb722050a4538b103418c6beb9347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:45.176793 15882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ebb722050a4538b103418c6beb9347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:45.179248 16230 tablet_bootstrap.cc:492] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: Bootstrap starting.
I20260927 11:15:45.179636 16150 tablet_service.cc:1467] Processing CreateTablet for tablet 86ebb722050a4538b103418c6beb9347 (DEFAULT_TABLE table=test-table [id=0c0f5ecb769541c9b641e0625ea244d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:15:45.179906 16230 tablet_bootstrap.cc:654] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:45.179917 16150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ebb722050a4538b103418c6beb9347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:15:45.180428 16230 log.cc:824] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:45.181088 16230 tablet_bootstrap.cc:492] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: No bootstrap required, opened a new log
I20260927 11:15:45.181170 16230 ts_tablet_manager.cc:1397] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:15:45.182366 16233 tablet_bootstrap.cc:492] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f: Bootstrap starting.
I20260927 11:15:45.182457 16231 tablet_bootstrap.cc:492] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6: Bootstrap starting.
I20260927 11:15:45.182847 16230 raft_consensus.cc:348] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } } peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } }
I20260927 11:15:45.183028 16230 raft_consensus.cc:374] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:45.183089 16233 tablet_bootstrap.cc:654] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:45.183072 16230 raft_consensus.cc:729] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d37f7a1207453b94b44ea2849bbabc, State: Initialized, Role: FOLLOWER
I20260927 11:15:45.183219 16231 tablet_bootstrap.cc:654] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6: Neither blocks nor log segments found. Creating new log.
I20260927 11:15:45.183267 16230 consensus_queue.cc:260] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } } peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } }
I20260927 11:15:45.183614 16230 ts_tablet_manager.cc:1428] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 11:15:45.183660 15948 heartbeater.cc:503] Master 127.0.98.62:36613 was elected leader, sending a full tablet report...
I20260927 11:15:45.183895 16231 log.cc:824] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:45.183986 16233 log.cc:824] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f: Log is configured to *not* fsync() on all Append() calls
I20260927 11:15:45.184628 16231 tablet_bootstrap.cc:492] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6: No bootstrap required, opened a new log
I20260927 11:15:45.184705 16231 ts_tablet_manager.cc:1397] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:15:45.186116 16231 raft_consensus.cc:348] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } } peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } }
I20260927 11:15:45.186295 16231 raft_consensus.cc:374] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:45.186350 16231 raft_consensus.cc:729] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7beba48c03a4d7fae7e832fa1384bd6, State: Initialized, Role: FOLLOWER
I20260927 11:15:45.186456 16231 consensus_queue.cc:260] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } } peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } }
I20260927 11:15:45.186543 16233 tablet_bootstrap.cc:492] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f: No bootstrap required, opened a new log
I20260927 11:15:45.186622 16233 ts_tablet_manager.cc:1397] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 11:15:45.186733 16231 ts_tablet_manager.cc:1428] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:15:45.186790 16082 heartbeater.cc:503] Master 127.0.98.62:36613 was elected leader, sending a full tablet report...
I20260927 11:15:45.188282 16233 raft_consensus.cc:348] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } } peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } }
I20260927 11:15:45.188407 16233 raft_consensus.cc:374] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:15:45.188428 16233 raft_consensus.cc:729] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c3c81a6cd1c49f6a05a952c5f81207f, State: Initialized, Role: FOLLOWER
I20260927 11:15:45.188529 16233 consensus_queue.cc:260] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } } peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } }
I20260927 11:15:45.188731 16233 ts_tablet_manager.cc:1428] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:15:45.188796 16216 heartbeater.cc:503] Master 127.0.98.62:36613 was elected leader, sending a full tablet report...
W20260927 11:15:45.230482 16083 tablet.cc:2369] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:15:45.339003 15949 tablet.cc:2369] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:15:45.392182 16217 tablet.cc:2369] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:15:45.550627 16237 raft_consensus.cc:482] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:15:45.550776 16237 raft_consensus.cc:504] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } } peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } }
I20260927 11:15:45.551151 16237 leader_election.cc:288] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c3c81a6cd1c49f6a05a952c5f81207f (127.0.98.3:40583), a9d37f7a1207453b94b44ea2849bbabc (127.0.98.1:40657)
I20260927 11:15:45.554008 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:45.551413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42657 (local address 127.0.98.1:40657)
0927 11:15:45.551473 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:45.551476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:45.551580 (+ 104us) server_negotiation.cc:407] Connection header received
0927 11:15:45.551624 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:45.551625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:45.551634 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:45.551701 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:45.551892 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.552320 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.553294 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.553649 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.553701 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:45.553708 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:45.553881 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 11:15:45.553924 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:15:45.554018 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:45.551385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57511 (local address 127.0.98.3:40583)
0927 11:15:45.551442 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:45.551444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:45.551561 (+ 117us) server_negotiation.cc:407] Connection header received
0927 11:15:45.551596 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:45.551597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:45.551611 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:45.551664 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:15:45.551831 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.552297 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.553289 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.553659 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.553746 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:45.553755 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:45.553913 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 11:15:45.553948 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 11:15:45.554092 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:45.551336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40657 (local address 127.0.98.2:42657)
0927 11:15:45.551476 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:45.551493 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:45.551571 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:45.551720 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:45.551721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:45.551722 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:45.551817 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:45.551820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.552438 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.552438 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:45.553199 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:45.553201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.553730 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.553731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:45.553808 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:45.553811 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:15:45.553853 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 11:15:45.553912 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":16,"thread_start_us":66,"threads_started":1}
I20260927 11:15:45.554108 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:45.551311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40583 (local address 127.0.98.2:57511)
0927 11:15:45.551452 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:45.551464 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:15:45.551534 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:15:45.551676 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:15:45.551677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:15:45.551679 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:15:45.551763 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:45.551764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.552367 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.552368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:45.553196 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:15:45.553198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.553758 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.553759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:15:45.553818 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:45.553822 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:15:45.553893 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:15:45.553928 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":16,"thread_start_us":67,"threads_started":1}
I20260927 11:15:45.554626 15902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ebb722050a4538b103418c6beb9347" candidate_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9d37f7a1207453b94b44ea2849bbabc" is_pre_election: true
I20260927 11:15:45.554652 16170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ebb722050a4538b103418c6beb9347" candidate_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" is_pre_election: true
I20260927 11:15:45.554768 15902 raft_consensus.cc:2454] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7beba48c03a4d7fae7e832fa1384bd6 in term 0.
I20260927 11:15:45.554778 16170 raft_consensus.cc:2454] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7beba48c03a4d7fae7e832fa1384bd6 in term 0.
I20260927 11:15:45.555001 15969 leader_election.cc:302] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c3c81a6cd1c49f6a05a952c5f81207f, a7beba48c03a4d7fae7e832fa1384bd6; no voters:
I20260927 11:15:45.555182 16237 raft_consensus.cc:2781] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:15:45.555228 16237 raft_consensus.cc:482] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:15:45.555243 16237 raft_consensus.cc:3037] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:45.555909 16237 raft_consensus.cc:504] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } } peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } }
I20260927 11:15:45.556036 16237 leader_election.cc:288] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [CANDIDATE]: Term 1 election: Requested vote from peers 5c3c81a6cd1c49f6a05a952c5f81207f (127.0.98.3:40583), a9d37f7a1207453b94b44ea2849bbabc (127.0.98.1:40657)
I20260927 11:15:45.556202 15902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ebb722050a4538b103418c6beb9347" candidate_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9d37f7a1207453b94b44ea2849bbabc"
I20260927 11:15:45.556208 16170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ebb722050a4538b103418c6beb9347" candidate_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f"
I20260927 11:15:45.556267 16170 raft_consensus.cc:3037] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:45.556288 15902 raft_consensus.cc:3037] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:15:45.556981 16170 raft_consensus.cc:2454] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7beba48c03a4d7fae7e832fa1384bd6 in term 1.
I20260927 11:15:45.557164 15969 leader_election.cc:302] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c3c81a6cd1c49f6a05a952c5f81207f, a7beba48c03a4d7fae7e832fa1384bd6; no voters:
I20260927 11:15:45.557163 15902 raft_consensus.cc:2454] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7beba48c03a4d7fae7e832fa1384bd6 in term 1.
I20260927 11:15:45.557305 16237 raft_consensus.cc:2781] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:15:45.557439 16237 raft_consensus.cc:686] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [term 1 LEADER]: Becoming Leader. State: Replica: a7beba48c03a4d7fae7e832fa1384bd6, State: Running, Role: LEADER
I20260927 11:15:45.557562 16237 consensus_queue.cc:237] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } } peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } }
I20260927 11:15:45.558331 15757 catalog_manager.cc:5495] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7beba48c03a4d7fae7e832fa1384bd6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } health_report { overall_health: HEALTHY } } }
I20260927 11:15:45.612730 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:45.610398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.0.98.2:40795)
0927 11:15:45.610452 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 11:15:45.610454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:15:45.610463 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:15:45.610501 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:15:45.610501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:15:45.610522 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:15:45.610600 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:15:45.610740 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.611195 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.611967 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:15:45.612178 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:15:45.612256 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:15:45.612421 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:15:45.612523 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:15:45.612613 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 11:15:45.612632 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 11:15:45.612652 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 11:15:45.614440 16170 raft_consensus.cc:1261] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f [term 1 FOLLOWER]: Refusing update from remote peer a7beba48c03a4d7fae7e832fa1384bd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:15:45.614737 15902 raft_consensus.cc:1261] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc [term 1 FOLLOWER]: Refusing update from remote peer a7beba48c03a4d7fae7e832fa1384bd6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:15:45.614771 16241 consensus_queue.cc:1035] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:15:45.614986 16241 consensus_queue.cc:1035] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:15:45.616853 16244 mvcc.cc:205] Tried to move back new op lower bound from 7333919726034251776 to 7333919725804417024. Current Snapshot: MvccSnapshot[applied={T|T < 7333919726034251776}]
I20260927 11:15:45.617865 16245 mvcc.cc:205] Tried to move back new op lower bound from 7333919726034251776 to 7333919725804417024. Current Snapshot: MvccSnapshot[applied={T|T < 7333919726034251776 or (T in {7333919726034251776})}]
I20260927 11:15:45.617921 16246 mvcc.cc:205] Tried to move back new op lower bound from 7333919726034251776 to 7333919725804417024. Current Snapshot: MvccSnapshot[applied={T|T < 7333919726034251776 or (T in {7333919726034251776})}]
I20260927 11:15:45.619944 15755 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:15:45.620052 15755 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:15:45.620793 15755 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Sending DeleteTablet for 3 replicas of tablet 86ebb722050a4538b103418c6beb9347
I20260927 11:15:45.621124 16016 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ebb722050a4538b103418c6beb9347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:15:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51576
I20260927 11:15:45.621330 16254 tablet_replica.cc:330] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6: stopping tablet replica
I20260927 11:15:45.621312 16150 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ebb722050a4538b103418c6beb9347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:15:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40228
I20260927 11:15:45.621433 16254 raft_consensus.cc:2229] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:15:45.621548 16254 raft_consensus.cc:2258] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:15:45.621634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 15819
I20260927 11:15:45.621657 15882 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ebb722050a4538b103418c6beb9347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:15:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56922
I20260927 11:15:45.622778 16255 tablet_replica.cc:330] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f: stopping tablet replica
I20260927 11:15:45.622920 16255 raft_consensus.cc:2229] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:15:45.623030 16255 raft_consensus.cc:2258] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:15:45.623045 16254 ts_tablet_manager.cc:1905] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:15:45.623450 16255 ts_tablet_manager.cc:1905] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:15:45.624994 16255 ts_tablet_manager.cc:1918] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:15:45.625082 16255 log.cc:1196] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/86ebb722050a4538b103418c6beb9347
I20260927 11:15:45.625097 16254 ts_tablet_manager.cc:1918] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:15:45.625187 16254 log.cc:1196] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/86ebb722050a4538b103418c6beb9347
I20260927 11:15:45.625346 16255 ts_tablet_manager.cc:1939] T 86ebb722050a4538b103418c6beb9347 P 5c3c81a6cd1c49f6a05a952c5f81207f: Deleting consensus metadata
I20260927 11:15:45.625481 16254 ts_tablet_manager.cc:1939] T 86ebb722050a4538b103418c6beb9347 P a7beba48c03a4d7fae7e832fa1384bd6: Deleting consensus metadata
I20260927 11:15:45.625782 15741 catalog_manager.cc:4841] TS 5c3c81a6cd1c49f6a05a952c5f81207f (127.0.98.3:40583): tablet 86ebb722050a4538b103418c6beb9347 (table test-table [id=0c0f5ecb769541c9b641e0625ea244d3]) successfully deleted
W20260927 11:15:45.627463 15743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40657 (error 108)
I20260927 11:15:45.627929 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:15:45.627803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40657 (local address 127.0.0.1:56926)
0927 11:15:45.627858 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 11:15:45.627897 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 11:15:45.628028 15744 catalog_manager.cc:4841] TS a7beba48c03a4d7fae7e832fa1384bd6 (127.0.98.2:40795): tablet 86ebb722050a4538b103418c6beb9347 (table test-table [id=0c0f5ecb769541c9b641e0625ea244d3]) successfully deleted
W20260927 11:15:45.628084 15743 catalog_manager.cc:4584] TS a9d37f7a1207453b94b44ea2849bbabc (127.0.98.1:40657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86ebb722050a4538b103418c6beb9347: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40657: connect: Connection refused (error 111)
I20260927 11:15:45.628203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 15953
I20260927 11:15:45.632637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 16087
I20260927 11:15:45.637210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 15725
I20260927 11:16:05.643383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36613
--webserver_interface=127.0.98.62
--webserver_port=45435
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:05.726948 16257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:05.727167 16257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:05.727205 16257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:05.728369 16257 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 11:16:05.728430 16257 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 11:16:05.728454 16257 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 11:16:05.728483 16257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:05.728511 16257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:05.728539 16257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:05.728564 16257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:05.728585 16257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:05.730007 16257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:05.730297 16257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:05.730525 16257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:05.740444 16266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:05.740473 16263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:05.740516 16257 server_base.cc:1034] running on GCE node
W20260927 11:16:05.740475 16264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:05.740901 16257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:05.741191 16257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:05.742344 16257 hybrid_clock.cc:648] HybridClock initialized: now 1790507765742330 us; error 41 us; skew 500 ppm
Sep 27 11:16:05 dist-test-slave-9hq9 krb5kdc[15712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:05.744671 16257 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:16:05.745108 16257 webserver.cc:460] Webserver started at http://127.0.98.62:45435/ using document root <none> and password file <none>
I20260927 11:16:05.745301 16257 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:05.745352 16257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:05.746671 16257 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:16:05.747246 16273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:05.747432 16257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:05.747496 16257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "f1a6b000a5af497db84cbd05c73d9437"
format_stamp: "Formatted at 2026-09-27 11:15:44 on dist-test-slave-9hq9"
server_key: "5688aaac04e53e6a6d5d8215b1ec06d9"
server_key_iv: "4979776320e4a40f6a55056e469ac6c5"
server_key_version: "encryptionkey@0"
I20260927 11:16:05.747841 16257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:05.755640 16257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:05.757172 16257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:05.757268 16257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:05.761426 16257 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36613
I20260927 11:16:05.761488 16325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36613 every 8 connection(s)
I20260927 11:16:05.761818 16257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:05.763195 16326 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:16:05.763747 16326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Bootstrap starting.
I20260927 11:16:05.768210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 16257
I20260927 11:16:05.768476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40657
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36613
--webserver_port=43733
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:16:05.771833 16326 log.cc:824] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:05.773510 16326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:16:05.773788 16326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Bootstrap complete.
I20260927 11:16:05.775992 16326 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 11:16:05.776357 16326 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a6b000a5af497db84cbd05c73d9437, State: Initialized, Role: FOLLOWER
I20260927 11:16:05.776535 16326 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 11:16:05.776614 16326 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:05.776722 16326 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:05.776804 16326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 1 FOLLOWER]: Advancing to term 2
I20260927 11:16:05.777721 16326 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 11:16:05.777803 16326 leader_election.cc:302] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1a6b000a5af497db84cbd05c73d9437; no voters:
I20260927 11:16:05.777956 16326 leader_election.cc:288] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 11:16:05.778013 16330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 11:16:05.778164 16330 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 2 LEADER]: Becoming Leader. State: Replica: f1a6b000a5af497db84cbd05c73d9437, State: Running, Role: LEADER
I20260927 11:16:05.778206 16326 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:05.778317 16330 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 11:16:05.778594 16331 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f1a6b000a5af497db84cbd05c73d9437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } } }
I20260927 11:16:05.778678 16331 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:05.778594 16332 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1a6b000a5af497db84cbd05c73d9437. Latest consensus state: current_term: 2 leader_uuid: "f1a6b000a5af497db84cbd05c73d9437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } } }
I20260927 11:16:05.778990 16335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:05.779006 16332 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:05.779708 16335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:16:05.779951 16335 catalog_manager.cc:1259] Loaded cluster ID: e790921a47884b6fb0b99b84418e1e5f
I20260927 11:16:05.780042 16335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:16:05.781232 16335 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:05.781390 16335 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Loaded TSK: 0
I20260927 11:16:05.790241 16335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Generated new TSK 1
I20260927 11:16:05.790331 16335 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 11:16:05.850440 16328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:05.850626 16328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:05.850648 16328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:05.851816 16328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:05.851856 16328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:05.851882 16328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:16:05.853243 16328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43733
--tserver_master_addrs=127.0.98.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:05.853440 16328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:05.853708 16328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:05.859438 16353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:05.859483 16354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:05.859622 16328 server_base.cc:1034] running on GCE node
W20260927 11:16:05.859483 16356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:05.859949 16328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:05.860214 16328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:05.861361 16328 hybrid_clock.cc:648] HybridClock initialized: now 1790507765861347 us; error 30 us; skew 500 ppm
Sep 27 11:16:05 dist-test-slave-9hq9 krb5kdc[15712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:05.863729 16328 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:16:05.864225 16328 webserver.cc:460] Webserver started at http://127.0.98.1:43733/ using document root <none> and password file <none>
I20260927 11:16:05.864435 16328 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:05.864482 16328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:05.865759 16328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:16:05.866483 16363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:05.866629 16328 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:05.866711 16328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "a9d37f7a1207453b94b44ea2849bbabc"
format_stamp: "Formatted at 2026-09-27 11:15:44 on dist-test-slave-9hq9"
server_key: "1440bebb20a1de3b6fe6727c21f2fa2f"
server_key_iv: "b1aedf4c1c453bfd29725f0a00a5e458"
server_key_version: "encryptionkey@0"
I20260927 11:16:05.866967 16328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:05.876441 16328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:05.878589 16328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:05.878753 16328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:05.878981 16328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:05.879448 16370 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 11:16:05.880759 16328 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 11:16:05.880808 16328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260927 11:16:05.880841 16328 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 11:16:05.881390 16328 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 11:16:05.881433 16328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 11:16:05.881472 16370 tablet_bootstrap.cc:492] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: Bootstrap starting.
I20260927 11:16:05.888993 16328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40657
I20260927 11:16:05.889498 16328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:16:05.890013 16477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40657 every 8 connection(s)
I20260927 11:16:05.892454 16370 log.cc:824] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:05.893152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 16328
I20260927 11:16:05.893450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40795
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36613
--webserver_port=46825
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:16:05.895641 16370 tablet_bootstrap.cc:492] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:16:05.895946 16370 tablet_bootstrap.cc:492] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: Bootstrap complete.
I20260927 11:16:05.897153 16370 ts_tablet_manager.cc:1397] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.010s
I20260927 11:16:05.898517 16370 raft_consensus.cc:348] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } } peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } }
I20260927 11:16:05.898813 16370 raft_consensus.cc:729] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d37f7a1207453b94b44ea2849bbabc, State: Initialized, Role: FOLLOWER
I20260927 11:16:05.898978 16370 consensus_queue.cc:260] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } } peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } }
I20260927 11:16:05.899248 16370 ts_tablet_manager.cc:1428] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
Sep 27 11:16:05 dist-test-slave-9hq9 krb5kdc[15712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:16:05.905179 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:05.890538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41723 (local address 127.0.98.62:36613)
0927 11:16:05.890740 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:05.890749 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:05.892834 (+ 2085us) server_negotiation.cc:407] Connection header received
0927 11:16:05.894851 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:05.894858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:05.894922 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:05.895054 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:05.896029 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:05.897458 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:05.898900 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:05.899459 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:05.899601 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:05.901736 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:05.901740 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:05.901740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:05.901752 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:05.903443 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:05.903901 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:05.903904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:05.903904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:05.903995 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:05.904269 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:05.904270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:05.904270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:05.904482 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:05.904571 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:05.904875 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 11:16:05.904998 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260927 11:16:05.905179 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:05.890175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:41723)
0927 11:16:05.892647 (+ 2472us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:05.892695 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:05.894572 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:05.895193 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:05.895197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:05.895544 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:05.895868 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:05.895874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:05.897589 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:05.897590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:05.898805 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:05.898807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:05.899571 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:05.899572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:05.899694 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:05.900295 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:05.900311 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:05.901632 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:05.903573 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:05.903580 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:05.903582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:05.903803 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:05.904085 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:05.904086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:05.904086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:05.904200 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:05.904571 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:05.904578 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:05.904707 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 11:16:05.904808 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 11:16:05.904980 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2428,"thread_start_us":90,"threads_started":1}
I20260927 11:16:05.905966 16478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36613
I20260927 11:16:05.906088 16478 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:05.906278 16478 heartbeater.cc:511] Master 127.0.98.62:36613 requested a full tablet report, sending...
I20260927 11:16:05.906975 16290 ts_manager.cc:194] Registered new tserver with Master: a9d37f7a1207453b94b44ea2849bbabc (127.0.98.1:40657)
I20260927 11:16:05.908305 16290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41723
I20260927 11:16:05.909191 16478 heartbeater.cc:503] Master 127.0.98.62:36613 was elected leader, sending a full tablet report...
Sep 27 11:16:05 dist-test-slave-9hq9 krb5kdc[15712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 11:16:05.915597 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:05.907733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52070 (local address 127.0.98.1:40657)
0927 11:16:05.907882 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:05.907886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:05.908127 (+ 241us) server_negotiation.cc:407] Connection header received
0927 11:16:05.908162 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:05.908164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:05.908178 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:05.908239 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:05.908906 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:05.909745 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:05.910565 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:05.910841 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:05.910929 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:05.912791 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:05.912798 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:05.912799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:05.912805 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:05.914366 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:05.914753 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:05.914756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:05.914757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:05.914816 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:05.915014 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:05.915015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:05.915016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:05.915152 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:05.915238 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:05.915423 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 11:16:05.915509 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20260927 11:16:05.915609 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:05.907528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40657 (local address 127.0.0.1:52070)
0927 11:16:05.907938 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:05.907969 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:05.908037 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:05.908293 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:05.908296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:05.908765 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:05.908816 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:05.908818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:05.909820 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:05.909822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:05.910464 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:05.910466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:05.910930 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:05.910933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:05.911034 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:05.911499 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:05.911503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:05.912703 (+ 1200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:05.914464 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:05.914470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:05.914472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:05.914670 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:05.914885 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:05.914886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:05.914886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:05.914950 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:05.915233 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:05.915238 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:05.915289 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 11:16:05.915389 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 11:16:05.915505 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":79,"threads_started":1}
I20260927 11:16:05.916023 16412 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ebb722050a4538b103418c6beb9347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:15:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52070
I20260927 11:16:05.916587 16487 tablet_replica.cc:330] stopping tablet replica
I20260927 11:16:05.916697 16487 raft_consensus.cc:2229] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:16:05.916782 16487 raft_consensus.cc:2258] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:05.917073 16487 ts_tablet_manager.cc:1905] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:16:05.919440 16487 ts_tablet_manager.cc:1918] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:16:05.919523 16487 log.cc:1196] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal/wals/86ebb722050a4538b103418c6beb9347
I20260927 11:16:05.919734 16487 ts_tablet_manager.cc:1939] T 86ebb722050a4538b103418c6beb9347 P a9d37f7a1207453b94b44ea2849bbabc: Deleting consensus metadata
I20260927 11:16:05.920218 16276 catalog_manager.cc:4841] TS a9d37f7a1207453b94b44ea2849bbabc (127.0.98.1:40657): tablet 86ebb722050a4538b103418c6beb9347 (table test-table [id=0c0f5ecb769541c9b641e0625ea244d3]) successfully deleted
W20260927 11:16:05.977871 16483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:05.978235 16483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:05.978287 16483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:05.979533 16483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:05.979591 16483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:05.979635 16483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:16:05.981036 16483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46825
--tserver_master_addrs=127.0.98.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:05.981295 16483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:05.981503 16483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:05.986287 16495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:05.986301 16492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:05.986582 16483 server_base.cc:1034] running on GCE node
W20260927 11:16:05.986304 16493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:05.986858 16483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:05.987093 16483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:05.988243 16483 hybrid_clock.cc:648] HybridClock initialized: now 1790507765988237 us; error 33 us; skew 500 ppm
Sep 27 11:16:05 dist-test-slave-9hq9 krb5kdc[15712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:05.990650 16483 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:16:05.990996 16483 webserver.cc:460] Webserver started at http://127.0.98.2:46825/ using document root <none> and password file <none>
I20260927 11:16:05.991199 16483 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:05.991250 16483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:05.992443 16483 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:16:05.992924 16502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:05.993076 16483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:05.993170 16483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "a7beba48c03a4d7fae7e832fa1384bd6"
format_stamp: "Formatted at 2026-09-27 11:15:44 on dist-test-slave-9hq9"
server_key: "fe46c923fbb4d83739d9a8ab1541ed6c"
server_key_iv: "5d094f06aa213a5f054fe2f5a1899068"
server_key_version: "encryptionkey@0"
I20260927 11:16:05.993462 16483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:06.006110 16483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:06.007752 16483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:06.007889 16483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:06.008087 16483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:06.008437 16483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:06.008469 16483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:06.008488 16483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:06.008502 16483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:06.014520 16483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40795
I20260927 11:16:06.014583 16615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40795 every 8 connection(s)
I20260927 11:16:06.014878 16483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:16:06.020187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 16483
Sep 27 11:16:06 dist-test-slave-9hq9 krb5kdc[15712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:16:06.020455 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40583
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36613
--webserver_port=42989
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:16:06.023941 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.015519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49437 (local address 127.0.98.62:36613)
0927 11:16:06.015577 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:06.015579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:06.015763 (+ 184us) server_negotiation.cc:407] Connection header received
0927 11:16:06.016204 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:06.016205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:06.016219 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:06.016265 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:06.017111 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.017590 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.018532 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.018783 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.018863 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.020955 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:06.020956 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:06.020956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:06.020962 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:06.022583 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:06.022976 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:06.022977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:06.022977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:06.023035 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:06.023284 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:06.023285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:06.023285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:06.023425 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:06.023493 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:06.023726 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 11:16:06.023820 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:16:06.023978 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.015520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:49437)
0927 11:16:06.015658 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:06.015687 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:06.016107 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:06.016338 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:06.016341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:06.016842 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:06.017009 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:06.017013 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.017678 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.017680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:06.018442 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:06.018444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.018873 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.018874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:06.018987 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.019440 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:06.019447 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:06.020873 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:06.022673 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:06.022679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:06.022680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:06.022905 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:06.023128 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:06.023128 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:06.023129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:06.023222 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:06.023487 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:06.023491 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:06.023592 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 11:16:06.023677 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 11:16:06.023784 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260927 11:16:06.024464 16616 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36613
I20260927 11:16:06.024559 16616 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:06.024744 16616 heartbeater.cc:511] Master 127.0.98.62:36613 requested a full tablet report, sending...
I20260927 11:16:06.025080 16290 ts_manager.cc:194] Registered new tserver with Master: a7beba48c03a4d7fae7e832fa1384bd6 (127.0.98.2:40795)
I20260927 11:16:06.025432 16290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49437
W20260927 11:16:06.098439 16619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:06.098616 16619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:06.098637 16619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:06.099709 16619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:06.099746 16619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:06.099771 16619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:16:06.101146 16619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42989
--tserver_master_addrs=127.0.98.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:06.101300 16619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:06.101505 16619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:06.106132 16627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:06.106160 16625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:06.106137 16624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:06.106267 16619 server_base.cc:1034] running on GCE node
I20260927 11:16:06.106516 16619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:06.106748 16619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:06.107879 16619 hybrid_clock.cc:648] HybridClock initialized: now 1790507766107859 us; error 35 us; skew 500 ppm
Sep 27 11:16:06 dist-test-slave-9hq9 krb5kdc[15712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:06.110221 16619 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:16:06.110632 16619 webserver.cc:460] Webserver started at http://127.0.98.3:42989/ using document root <none> and password file <none>
I20260927 11:16:06.110805 16619 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:06.110854 16619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:06.112001 16619 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:16:06.112484 16634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:06.112636 16619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:06.112723 16619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "5c3c81a6cd1c49f6a05a952c5f81207f"
format_stamp: "Formatted at 2026-09-27 11:15:45 on dist-test-slave-9hq9"
server_key: "a2126f507d59181b2c150a2052a99e04"
server_key_iv: "9c82276b27709e9741832ba456c5a4af"
server_key_version: "encryptionkey@0"
I20260927 11:16:06.112999 16619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:06.124440 16619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:06.126121 16619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:06.126259 16619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:06.126451 16619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:06.126716 16619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:06.126766 16619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:06.126798 16619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:06.126823 16619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:06.132521 16619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40583
I20260927 11:16:06.132555 16747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40583 every 8 connection(s)
I20260927 11:16:06.132861 16619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:16:06.135499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 16619
Sep 27 11:16:06 dist-test-slave-9hq9 krb5kdc[15712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:16:06.142185 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.133488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58153 (local address 127.0.98.62:36613)
0927 11:16:06.133530 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:06.133532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:06.133745 (+ 213us) server_negotiation.cc:407] Connection header received
0927 11:16:06.134197 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:06.134198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:06.134207 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:06.134257 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:06.135045 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.135536 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.136840 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.137024 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.137080 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.139228 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:06.139229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:06.139229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:06.139235 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:06.140750 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:06.141132 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:06.141133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:06.141134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:06.141188 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:06.141459 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:06.141460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:06.141460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:06.141618 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:06.141682 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:06.141983 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 11:16:06.142084 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:16:06.142305 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.135899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.0.98.62:36613)
0927 11:16:06.136017 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:06.136018 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:06.136088 (+ 70us) server_negotiation.cc:407] Connection header received
0927 11:16:06.136161 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:06.136162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:06.136174 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:06.136243 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:06.136784 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.137430 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.138290 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.138485 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.138535 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.139968 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:06.139969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:06.139969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:06.139974 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:06.141189 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:06.141503 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:06.141503 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:06.141504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:06.141547 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:06.141752 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:06.141753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:06.141753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:06.141858 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:06.142029 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:06.142085 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 11:16:06.142152 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260927 11:16:06.142592 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.133460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:58153)
0927 11:16:06.133606 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:06.133634 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:06.134102 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:06.134303 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:06.134305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:06.134797 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:06.134958 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:06.134962 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.135758 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.135759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:06.136756 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:06.136760 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.137107 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.137108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:06.137242 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.137745 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:06.137753 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:06.139156 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:06.140833 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:06.140837 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:06.140839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:06.141016 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:06.141298 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:06.141299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:06.141299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:06.141397 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:06.141707 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:06.141712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:06.141857 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 11:16:06.141955 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 11:16:06.142116 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260927 11:16:06.143050 16748 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36613
I20260927 11:16:06.143152 16748 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:06.143292 16748 heartbeater.cc:511] Master 127.0.98.62:36613 requested a full tablet report, sending...
I20260927 11:16:06.143586 16290 ts_manager.cc:194] Registered new tserver with Master: 5c3c81a6cd1c49f6a05a952c5f81207f (127.0.98.3:40583)
I20260927 11:16:06.143947 16290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58153
I20260927 11:16:06.153795 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:16:06.156087 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.154119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56514 (local address 127.0.98.62:36613)
0927 11:16:06.154187 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:06.154188 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:06.154237 (+ 49us) server_negotiation.cc:407] Connection header received
0927 11:16:06.154307 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:06.154308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:06.154333 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:06.154387 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:16:06.154476 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.154897 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.155570 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.155737 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.155782 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.155894 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:16:06.155928 (+ 34us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 11:16:06.156025 (+ 97us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56514: token expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 11:16:06.156137 16752 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56514: token expired
W20260927 11:16:06.156229 16222 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:36613)
I20260927 11:16:06.161332 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.156459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56522 (local address 127.0.98.62:36613)
0927 11:16:06.156490 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:06.156491 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:06.156530 (+ 39us) server_negotiation.cc:407] Connection header received
0927 11:16:06.156582 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:06.156583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:06.156589 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:06.156626 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:06.157008 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.157404 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.158088 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.158271 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.158323 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.159282 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:06.159283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:06.159283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:06.159288 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:06.160507 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:06.160789 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:06.160789 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:06.160789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:06.160835 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:06.160976 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:06.160976 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:06.160977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:06.161052 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:06.161115 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:06.161197 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 11:16:06.161250 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 11:16:06.162273 16290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:16:06.163774 16290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:16:06.165803 16412 tablet_service.cc:1467] Processing CreateTablet for tablet 1efaf5ac7c584fe7b598fa5ba85ec60e (DEFAULT_TABLE table=test-table [id=d6764f88c866477898cb665301831e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:06.166002 16412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1efaf5ac7c584fe7b598fa5ba85ec60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:06.167305 16370 tablet_bootstrap.cc:492] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc: Bootstrap starting.
I20260927 11:16:06.167599 16370 tablet_bootstrap.cc:654] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:06.168459 16370 tablet_bootstrap.cc:492] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc: No bootstrap required, opened a new log
I20260927 11:16:06.168509 16370 ts_tablet_manager.cc:1397] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 11:16:06.168645 16370 raft_consensus.cc:348] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:06.168744 16370 raft_consensus.cc:374] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:06.168802 16370 raft_consensus.cc:729] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d37f7a1207453b94b44ea2849bbabc, State: Initialized, Role: FOLLOWER
I20260927 11:16:06.168852 16370 consensus_queue.cc:260] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:06.168941 16370 ts_tablet_manager.cc:1428] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:06.169757 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.165532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40795 (local address 127.0.0.1:59134)
0927 11:16:06.165562 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:06.165573 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:06.165611 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:06.166376 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:06.166377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:06.166378 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:06.166404 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:06.166418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.167541 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.167542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:06.168470 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:06.168473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.169513 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.169514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:06.169578 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.169596 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 11:16:06.169648 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 11:16:06.169693 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17,"mutex_wait_us":3}
I20260927 11:16:06.169828 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.165613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40583 (local address 127.0.0.1:44654)
0927 11:16:06.165761 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:06.165770 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:06.165808 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:06.166206 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:06.166207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:06.166209 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:06.166251 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:06.166252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.167476 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.167477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:06.168645 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:06.168648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.169552 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.169552 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:06.169610 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.169613 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:06.169692 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 11:16:06.169728 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260927 11:16:06.169899 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.165848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.0.98.3:40583)
0927 11:16:06.165991 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:06.165996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:06.166009 (+ 13us) server_negotiation.cc:407] Connection header received
0927 11:16:06.166034 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:06.166035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:06.166079 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:06.166196 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:06.166316 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.167385 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.168728 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.169477 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.169548 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.169597 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:06.169705 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 11:16:06.169742 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260927 11:16:06.169801 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.165666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59134 (local address 127.0.98.2:40795)
0927 11:16:06.166203 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:06.166208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:06.166217 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:16:06.166255 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:06.166257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:06.166273 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:06.166378 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:06.166461 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.167484 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.168586 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.169327 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.169408 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.169460 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:06.169676 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 11:16:06.169718 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":90,"threads_started":1}
I20260927 11:16:06.170533 16682 tablet_service.cc:1467] Processing CreateTablet for tablet 1efaf5ac7c584fe7b598fa5ba85ec60e (DEFAULT_TABLE table=test-table [id=d6764f88c866477898cb665301831e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:06.170835 16682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1efaf5ac7c584fe7b598fa5ba85ec60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:06.170753 16550 tablet_service.cc:1467] Processing CreateTablet for tablet 1efaf5ac7c584fe7b598fa5ba85ec60e (DEFAULT_TABLE table=test-table [id=d6764f88c866477898cb665301831e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:06.171056 16550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1efaf5ac7c584fe7b598fa5ba85ec60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:06.174199 16758 tablet_bootstrap.cc:492] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6: Bootstrap starting.
I20260927 11:16:06.174199 16759 tablet_bootstrap.cc:492] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: Bootstrap starting.
I20260927 11:16:06.174861 16759 tablet_bootstrap.cc:654] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:06.174861 16758 tablet_bootstrap.cc:654] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:06.175412 16759 log.cc:824] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:06.175468 16758 log.cc:824] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:06.176148 16759 tablet_bootstrap.cc:492] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: No bootstrap required, opened a new log
I20260927 11:16:06.176216 16759 ts_tablet_manager.cc:1397] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:16:06.176294 16758 tablet_bootstrap.cc:492] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6: No bootstrap required, opened a new log
I20260927 11:16:06.176353 16758 ts_tablet_manager.cc:1397] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 11:16:06.178097 16759 raft_consensus.cc:348] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:06.178148 16758 raft_consensus.cc:348] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:06.178274 16759 raft_consensus.cc:374] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:06.178279 16758 raft_consensus.cc:374] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:06.178313 16758 raft_consensus.cc:729] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7beba48c03a4d7fae7e832fa1384bd6, State: Initialized, Role: FOLLOWER
I20260927 11:16:06.178313 16759 raft_consensus.cc:729] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c3c81a6cd1c49f6a05a952c5f81207f, State: Initialized, Role: FOLLOWER
I20260927 11:16:06.178429 16758 consensus_queue.cc:260] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:06.178429 16759 consensus_queue.cc:260] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:06.178735 16759 ts_tablet_manager.cc:1428] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:16:06.178735 16758 ts_tablet_manager.cc:1428] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 11:16:06.178771 16748 heartbeater.cc:503] Master 127.0.98.62:36613 was elected leader, sending a full tablet report...
I20260927 11:16:06.178850 16616 heartbeater.cc:503] Master 127.0.98.62:36613 was elected leader, sending a full tablet report...
I20260927 11:16:06.221339 16763 raft_consensus.cc:482] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:16:06.221433 16763 raft_consensus.cc:504] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:06.221803 16763 leader_election.cc:288] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7beba48c03a4d7fae7e832fa1384bd6 (127.0.98.2:40795), a9d37f7a1207453b94b44ea2849bbabc (127.0.98.1:40657)
I20260927 11:16:06.224567 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.221953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40795 (local address 127.0.98.3:53899)
0927 11:16:06.222001 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:06.222013 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:06.222087 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:06.222250 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:06.222252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:06.222253 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:06.222323 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:06.222325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.222963 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.222964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:06.223733 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:06.223735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.224311 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.224312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:06.224399 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.224403 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:16:06.224466 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 11:16:06.224500 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 11:16:06.224593 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.222013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53899 (local address 127.0.98.2:40795)
0927 11:16:06.222069 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:06.222071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:06.222080 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:16:06.222162 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:06.222163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:06.222174 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:06.222235 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:06.222394 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.222893 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.223826 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.224236 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.224286 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.224294 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:06.224494 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 11:16:06.224527 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:16:06.224776 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.222023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35565 (local address 127.0.98.1:40657)
0927 11:16:06.222078 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:06.222080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:06.222245 (+ 165us) server_negotiation.cc:407] Connection header received
0927 11:16:06.222274 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:06.222275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:06.222299 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:06.222367 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:06.222539 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.223020 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.223950 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.224414 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.224475 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.224551 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:06.224645 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 11:16:06.224694 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:16:06.224929 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1efaf5ac7c584fe7b598fa5ba85ec60e" candidate_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" is_pre_election: true
I20260927 11:16:06.224980 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.221969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40657 (local address 127.0.98.3:35565)
0927 11:16:06.222147 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:06.222157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:06.222228 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:06.222393 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:06.222394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:06.222395 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:06.222456 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:06.222458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.223115 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.223116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:06.223865 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:06.223866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.224499 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.224500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:06.224566 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.224569 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:06.224627 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 11:16:06.224667 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":107,"threads_started":1}
I20260927 11:16:06.225080 16570 raft_consensus.cc:2454] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c3c81a6cd1c49f6a05a952c5f81207f in term 0.
I20260927 11:16:06.225330 16638 leader_election.cc:302] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c3c81a6cd1c49f6a05a952c5f81207f, a7beba48c03a4d7fae7e832fa1384bd6; no voters:
I20260927 11:16:06.225306 16432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1efaf5ac7c584fe7b598fa5ba85ec60e" candidate_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9d37f7a1207453b94b44ea2849bbabc" is_pre_election: true
I20260927 11:16:06.225447 16432 raft_consensus.cc:2454] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c3c81a6cd1c49f6a05a952c5f81207f in term 0.
I20260927 11:16:06.225449 16763 raft_consensus.cc:2781] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:16:06.225517 16763 raft_consensus.cc:482] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:16:06.225556 16763 raft_consensus.cc:3037] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:06.226203 16763 raft_consensus.cc:504] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:06.226325 16763 leader_election.cc:288] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [CANDIDATE]: Term 1 election: Requested vote from peers a7beba48c03a4d7fae7e832fa1384bd6 (127.0.98.2:40795), a9d37f7a1207453b94b44ea2849bbabc (127.0.98.1:40657)
I20260927 11:16:06.226468 16432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1efaf5ac7c584fe7b598fa5ba85ec60e" candidate_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9d37f7a1207453b94b44ea2849bbabc"
I20260927 11:16:06.226455 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1efaf5ac7c584fe7b598fa5ba85ec60e" candidate_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7beba48c03a4d7fae7e832fa1384bd6"
I20260927 11:16:06.226534 16432 raft_consensus.cc:3037] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:06.226534 16570 raft_consensus.cc:3037] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:06.227355 16432 raft_consensus.cc:2454] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c3c81a6cd1c49f6a05a952c5f81207f in term 1.
I20260927 11:16:06.227355 16570 raft_consensus.cc:2454] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c3c81a6cd1c49f6a05a952c5f81207f in term 1.
I20260927 11:16:06.227524 16637 leader_election.cc:302] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c3c81a6cd1c49f6a05a952c5f81207f, a9d37f7a1207453b94b44ea2849bbabc; no voters:
I20260927 11:16:06.227622 16763 raft_consensus.cc:2781] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:06.227779 16763 raft_consensus.cc:686] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 1 LEADER]: Becoming Leader. State: Replica: 5c3c81a6cd1c49f6a05a952c5f81207f, State: Running, Role: LEADER
I20260927 11:16:06.227871 16763 consensus_queue.cc:237] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:06.228684 16289 catalog_manager.cc:5495] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c3c81a6cd1c49f6a05a952c5f81207f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } health_report { overall_health: UNKNOWN } } }
I20260927 11:16:06.244037 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.241952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44664 (local address 127.0.98.3:40583)
0927 11:16:06.242004 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:06.242006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:06.242021 (+ 15us) server_negotiation.cc:407] Connection header received
0927 11:16:06.242116 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:06.242117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:06.242132 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:06.242190 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:16:06.242295 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.242757 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.243423 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:06.243590 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:06.243637 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:06.243754 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:16:06.243831 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:16:06.243887 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:06.243919 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 11:16:06.243934 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:16:06.245460 16570 raft_consensus.cc:1261] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 1 FOLLOWER]: Refusing update from remote peer 5c3c81a6cd1c49f6a05a952c5f81207f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:16:06.245729 16432 raft_consensus.cc:1261] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc [term 1 FOLLOWER]: Refusing update from remote peer 5c3c81a6cd1c49f6a05a952c5f81207f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:16:06.245734 16763 consensus_queue.cc:1035] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:16:06.245960 16763 consensus_queue.cc:1035] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:16:06.248102 16768 mvcc.cc:205] Tried to move back new op lower bound from 7333919810539417600 to 7333919810470203392. Current Snapshot: MvccSnapshot[applied={T|T < 7333919810539417600}]
I20260927 11:16:06.248615 16769 mvcc.cc:205] Tried to move back new op lower bound from 7333919810539417600 to 7333919810470203392. Current Snapshot: MvccSnapshot[applied={T|T < 7333919810539417600}]
I20260927 11:16:06.250021 16771 mvcc.cc:205] Tried to move back new op lower bound from 7333919810539417600 to 7333919810470203392. Current Snapshot: MvccSnapshot[applied={T|T < 7333919810539417600 or (T in {7333919810539417600})}]
I20260927 11:16:06.250345 16289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:16:06.250423 16289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:16:06.251132 16289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Sending DeleteTablet for 3 replicas of tablet 1efaf5ac7c584fe7b598fa5ba85ec60e
I20260927 11:16:06.251431 16550 tablet_service.cc:1514] Processing DeleteTablet for tablet 1efaf5ac7c584fe7b598fa5ba85ec60e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59134
I20260927 11:16:06.251448 16682 tablet_service.cc:1514] Processing DeleteTablet for tablet 1efaf5ac7c584fe7b598fa5ba85ec60e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44654
I20260927 11:16:06.251531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 16328
W20260927 11:16:06.257256 16276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40657 (error 108)
I20260927 11:16:06.257431 16778 tablet_replica.cc:330] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: stopping tablet replica
I20260927 11:16:06.257558 16777 tablet_replica.cc:330] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6: stopping tablet replica
I20260927 11:16:06.257594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 16483
I20260927 11:16:06.257650 16777 raft_consensus.cc:2229] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:16:06.257526 16778 raft_consensus.cc:2229] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:16:06.258867 16778 raft_consensus.cc:2258] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:06.260097 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.259987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40657 (local address 127.0.0.1:52086)
0927 11:16:06.260029 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:06.260063 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 11:16:06.260179 16778 ts_tablet_manager.cc:1905] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 11:16:06.260253 16276 catalog_manager.cc:4584] TS a9d37f7a1207453b94b44ea2849bbabc (127.0.98.1:40657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1efaf5ac7c584fe7b598fa5ba85ec60e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40657: connect: Connection refused (error 111)
I20260927 11:16:06.261837 16778 ts_tablet_manager.cc:1918] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:16:06.261904 16778 log.cc:1196] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/1efaf5ac7c584fe7b598fa5ba85ec60e
I20260927 11:16:06.262185 16778 ts_tablet_manager.cc:1939] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: Deleting consensus metadata
I20260927 11:16:06.262547 16274 catalog_manager.cc:4841] TS 5c3c81a6cd1c49f6a05a952c5f81207f (127.0.98.3:40583): tablet 1efaf5ac7c584fe7b598fa5ba85ec60e (table test-table [id=d6764f88c866477898cb665301831e94]) successfully deleted
I20260927 11:16:06.262871 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:06.262765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40795 (local address 127.0.0.1:59140)
0927 11:16:06.262817 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:06.262847 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 11:16:06.262975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 16619
I20260927 11:16:06.267583 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 16257
I20260927 11:16:26.272830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36613
--webserver_interface=127.0.98.62
--webserver_port=45435
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:26.348524 16780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:26.348724 16780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:26.348760 16780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:26.349892 16780 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 11:16:26.349951 16780 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 11:16:26.349974 16780 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 11:16:26.350004 16780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:26.350028 16780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:26.350081 16780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:26.350106 16780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:26.350126 16780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:26.351470 16780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:26.351639 16780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:26.351778 16780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:26.356156 16786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:26.356182 16788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:26.356215 16780 server_base.cc:1034] running on GCE node
W20260927 11:16:26.356180 16785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:26.356539 16780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:26.356846 16780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:26.358009 16780 hybrid_clock.cc:648] HybridClock initialized: now 1790507786358007 us; error 34 us; skew 500 ppm
Sep 27 11:16:26 dist-test-slave-9hq9 krb5kdc[15712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:26.360515 16780 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:16:26.360868 16780 webserver.cc:460] Webserver started at http://127.0.98.62:45435/ using document root <none> and password file <none>
I20260927 11:16:26.361030 16780 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:26.361068 16780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:26.362197 16780 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:16:26.362772 16795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:26.362946 16780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:26.363035 16780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "f1a6b000a5af497db84cbd05c73d9437"
format_stamp: "Formatted at 2026-09-27 11:15:44 on dist-test-slave-9hq9"
server_key: "5688aaac04e53e6a6d5d8215b1ec06d9"
server_key_iv: "4979776320e4a40f6a55056e469ac6c5"
server_key_version: "encryptionkey@0"
I20260927 11:16:26.363343 16780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:26.386395 16780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:26.388005 16780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:26.388154 16780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:26.392066 16780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36613
I20260927 11:16:26.392097 16847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36613 every 8 connection(s)
I20260927 11:16:26.392444 16780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:26.393757 16848 sys_catalog.cc:263] Verifying existing consensus state
I20260927 11:16:26.394484 16848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Bootstrap starting.
I20260927 11:16:26.397257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 16780
I20260927 11:16:26.397567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40657
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36613
--webserver_port=43733
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:16:26.403206 16848 log.cc:824] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:26.405248 16848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:16:26.405530 16848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Bootstrap complete.
I20260927 11:16:26.407614 16848 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 11:16:26.407836 16848 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a6b000a5af497db84cbd05c73d9437, State: Initialized, Role: FOLLOWER
I20260927 11:16:26.408008 16848 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 11:16:26.408082 16848 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:26.408130 16848 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:26.408202 16848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 2 FOLLOWER]: Advancing to term 3
I20260927 11:16:26.409108 16848 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 11:16:26.409251 16848 leader_election.cc:302] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1a6b000a5af497db84cbd05c73d9437; no voters:
I20260927 11:16:26.409431 16848 leader_election.cc:288] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 11:16:26.409473 16852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 11:16:26.409614 16852 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [term 3 LEADER]: Becoming Leader. State: Replica: f1a6b000a5af497db84cbd05c73d9437, State: Running, Role: LEADER
I20260927 11:16:26.409685 16848 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:26.409698 16852 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } }
I20260927 11:16:26.410033 16852 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1a6b000a5af497db84cbd05c73d9437. Latest consensus state: current_term: 3 leader_uuid: "f1a6b000a5af497db84cbd05c73d9437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } } }
I20260927 11:16:26.410121 16852 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:26.410307 16857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:26.410425 16853 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f1a6b000a5af497db84cbd05c73d9437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1a6b000a5af497db84cbd05c73d9437" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36613 } } }
I20260927 11:16:26.410554 16853 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:26.411036 16857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:16:26.411134 16857 catalog_manager.cc:1259] Loaded cluster ID: e790921a47884b6fb0b99b84418e1e5f
I20260927 11:16:26.411167 16857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:16:26.412523 16857 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:26.412704 16857 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Loaded TSK: 0
I20260927 11:16:26.412752 16857 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f1a6b000a5af497db84cbd05c73d9437: Loaded TSK: 1
F20260927 11:16:26.412964 16857 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f566375b9ad google::LogMessage::Fail() at ??:0
@ 0x7f566375e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f566375b4de google::LogMessage::Flush() at ??:0
@ 0x7f566375ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5665ec8104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f5665ec5d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f566b6b51cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f566b6b548e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f566b6b5594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f566b67bc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f566b685ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f56656290d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f566561e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f56664826db start_thread at ??:0
@ 0x7f566225961f clone at ??:0
W20260927 11:16:26.493610 16850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:26.493825 16850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:26.493861 16850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:26.495052 16850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:26.495115 16850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:26.495160 16850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:16:26.496572 16850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43733
--tserver_master_addrs=127.0.98.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:26.496820 16850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:26.497040 16850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:26.501701 16892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:26.501725 16890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:26.501819 16850 server_base.cc:1034] running on GCE node
W20260927 11:16:26.501876 16889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:26.502142 16850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:26.502408 16850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:26.503557 16850 hybrid_clock.cc:648] HybridClock initialized: now 1790507786503544 us; error 31 us; skew 500 ppm
Sep 27 11:16:26 dist-test-slave-9hq9 krb5kdc[15712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:26.505981 16850 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:16:26.506455 16850 webserver.cc:460] Webserver started at http://127.0.98.1:43733/ using document root <none> and password file <none>
I20260927 11:16:26.506683 16850 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:26.506726 16850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:26.507979 16850 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:16:26.508658 16899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:26.508800 16850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:26.508884 16850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "a9d37f7a1207453b94b44ea2849bbabc"
format_stamp: "Formatted at 2026-09-27 11:15:44 on dist-test-slave-9hq9"
server_key: "1440bebb20a1de3b6fe6727c21f2fa2f"
server_key_iv: "b1aedf4c1c453bfd29725f0a00a5e458"
server_key_version: "encryptionkey@0"
I20260927 11:16:26.509198 16850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:26.523552 16850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:26.525571 16850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:26.525672 16850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:26.525954 16850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:26.526517 16906 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 11:16:26.528360 16850 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 11:16:26.528419 16850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 11:16:26.528460 16850 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 11:16:26.529033 16850 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 11:16:26.529086 16850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 11:16:26.529176 16906 tablet_bootstrap.cc:492] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc: Bootstrap starting.
I20260927 11:16:26.536154 16850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40657
I20260927 11:16:26.536193 17013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40657 every 8 connection(s)
I20260927 11:16:26.536542 16850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:16:26.539999 16906 log.cc:824] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:26.542551 16906 tablet_bootstrap.cc:492] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:16:26.542595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 16850
I20260927 11:16:26.542862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40795
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36613
--webserver_port=46825
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:16:26.542873 16906 tablet_bootstrap.cc:492] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc: Bootstrap complete.
I20260927 11:16:26.544086 16906 ts_tablet_manager.cc:1397] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.011s
I20260927 11:16:26.545464 16906 raft_consensus.cc:348] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:26.545794 16906 raft_consensus.cc:729] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d37f7a1207453b94b44ea2849bbabc, State: Initialized, Role: FOLLOWER
I20260927 11:16:26.545959 16906 consensus_queue.cc:260] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:26.546263 16906 ts_tablet_manager.cc:1428] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
W20260927 11:16:26.606513 17016 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:16:26.537248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:38327)
0927 11:16:26.538187 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:26.538221 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:26.538633 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:26.606361 (+ 67728us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":896,"thread_start_us":71,"threads_started":1}
W20260927 11:16:26.606796 17014 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36613 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36613: Client connection negotiation failed: client connection to 127.0.98.62:36613: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 11:16:26.607146 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:26.607031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:37949)
0927 11:16:26.607082 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:26.607126 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 11:16:26.607486 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:26.607391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:36163)
0927 11:16:26.607440 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:26.607466 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 11:16:26.607601 17014 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 11:16:26.622603 17018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:26.622809 17018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:26.622845 17018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:26.623961 17018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:26.624023 17018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:26.624066 17018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:16:26.625458 17018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46825
--tserver_master_addrs=127.0.98.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:26.625692 17018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:26.625902 17018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:26.630450 17025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:26.630501 17027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:26.630486 17024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:26.630647 17018 server_base.cc:1034] running on GCE node
I20260927 11:16:26.630841 17018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:26.631074 17018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:26.632216 17018 hybrid_clock.cc:648] HybridClock initialized: now 1790507786632192 us; error 41 us; skew 500 ppm
Sep 27 11:16:26 dist-test-slave-9hq9 krb5kdc[15712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:26.634542 17018 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:16:26.634966 17018 webserver.cc:460] Webserver started at http://127.0.98.2:46825/ using document root <none> and password file <none>
I20260927 11:16:26.635129 17018 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:26.635169 17018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:26.636277 17018 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:16:26.636809 17034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:26.636969 17018 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:26.637060 17018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "a7beba48c03a4d7fae7e832fa1384bd6"
format_stamp: "Formatted at 2026-09-27 11:15:44 on dist-test-slave-9hq9"
server_key: "fe46c923fbb4d83739d9a8ab1541ed6c"
server_key_iv: "5d094f06aa213a5f054fe2f5a1899068"
server_key_version: "encryptionkey@0"
I20260927 11:16:26.637362 17018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:26.664628 17018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:26.666477 17018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:26.666639 17018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:26.666873 17018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:26.667306 17041 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 11:16:26.668605 17018 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 11:16:26.668679 17018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260927 11:16:26.668718 17018 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 11:16:26.669395 17018 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 11:16:26.669448 17018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 11:16:26.669477 17041 tablet_bootstrap.cc:492] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6: Bootstrap starting.
I20260927 11:16:26.676525 17018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40795
I20260927 11:16:26.676620 17148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40795 every 8 connection(s)
I20260927 11:16:26.676924 17018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:16:26.677938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 17018
I20260927 11:16:26.678094 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:26.677579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:37221)
0927 11:16:26.677785 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:26.677900 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
I20260927 11:16:26.678261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40583
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36613
--webserver_port=42989
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:26.678318 17149 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36613 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36613: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
I20260927 11:16:26.678548 17041 log.cc:824] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:26.678716 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:26.678486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:55399)
0927 11:16:26.678600 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:26.678671 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
I20260927 11:16:26.679064 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:26.678961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:42013)
0927 11:16:26.679009 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:26.679035 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 11:16:26.679198 17149 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 11:16:26.681138 17041 tablet_bootstrap.cc:492] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:16:26.681412 17041 tablet_bootstrap.cc:492] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6: Bootstrap complete.
I20260927 11:16:26.682538 17041 ts_tablet_manager.cc:1397] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.006s
I20260927 11:16:26.684713 17041 raft_consensus.cc:348] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:26.685134 17041 raft_consensus.cc:729] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7beba48c03a4d7fae7e832fa1384bd6, State: Initialized, Role: FOLLOWER
I20260927 11:16:26.685283 17041 consensus_queue.cc:260] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:26.685570 17041 ts_tablet_manager.cc:1428] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 11:16:26.685747 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:26.685669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:53047)
0927 11:16:26.685700 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:26.685724 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260927 11:16:26.686123 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:26.685995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:60765)
0927 11:16:26.686037 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:26.686105 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 11:16:26.759361 17154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:26.759625 17154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:26.759649 17154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:26.760788 17154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:26.760828 17154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:26.760854 17154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:16:26.762210 17154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42989
--tserver_master_addrs=127.0.98.62:36613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:26.762378 17154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:26.762542 17154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:26.767184 17163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:26.767180 17160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:26.767205 17161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:26.767409 17154 server_base.cc:1034] running on GCE node
I20260927 11:16:26.767629 17154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:26.767822 17154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:26.768970 17154 hybrid_clock.cc:648] HybridClock initialized: now 1790507786768956 us; error 27 us; skew 500 ppm
Sep 27 11:16:26 dist-test-slave-9hq9 krb5kdc[15712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:26.771411 17154 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:16:26.771880 17154 webserver.cc:460] Webserver started at http://127.0.98.3:42989/ using document root <none> and password file <none>
I20260927 11:16:26.772115 17154 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:26.772183 17154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:26.773450 17154 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:16:26.774080 17170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:26.774251 17154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:16:26.774348 17154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "5c3c81a6cd1c49f6a05a952c5f81207f"
format_stamp: "Formatted at 2026-09-27 11:15:45 on dist-test-slave-9hq9"
server_key: "a2126f507d59181b2c150a2052a99e04"
server_key_iv: "9c82276b27709e9741832ba456c5a4af"
server_key_version: "encryptionkey@0"
I20260927 11:16:26.774670 17154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:26.785840 17154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:26.788132 17154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:26.788286 17154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:26.788494 17154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:26.788784 17154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:26.788834 17154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:26.788873 17154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:26.788899 17154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:26.794564 17154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40583
I20260927 11:16:26.794607 17283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40583 every 8 connection(s)
I20260927 11:16:26.794929 17154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:16:26.795903 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:26.795561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:56647)
0927 11:16:26.795702 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:26.795773 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
W20260927 11:16:26.796269 17284 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36613 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36613: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
I20260927 11:16:26.796535 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:26.796451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:55993)
0927 11:16:26.796494 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:26.796519 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 11:16:26.796779 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:26.796723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:47831)
0927 11:16:26.796744 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:26.796768 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260927 11:16:26.796888 17284 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 11:16:26.804231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 17154
I20260927 11:16:27.608209 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:27.607933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:46929)
0927 11:16:27.608120 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:27.608152 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":54,"threads_started":1}
I20260927 11:16:27.686690 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:27.686505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:59887)
0927 11:16:27.686629 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:27.686658 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260927 11:16:27.797346 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:27.797166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:41305)
0927 11:16:27.797273 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:27.797302 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":52,"threads_started":1}
I20260927 11:16:28.123375 17291 raft_consensus.cc:482] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:16:28.123526 17291 raft_consensus.cc:504] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:28.123862 17291 leader_election.cc:288] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c3c81a6cd1c49f6a05a952c5f81207f (127.0.98.3:40583), a9d37f7a1207453b94b44ea2849bbabc (127.0.98.1:40657)
Sep 27 11:16:28 dist-test-slave-9hq9 krb5kdc[15712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
Sep 27 11:16:28 dist-test-slave-9hq9 krb5kdc[15712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 11:16:28.133389 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:28.124123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48315 (local address 127.0.98.1:40657)
0927 11:16:28.124279 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:28.124286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:28.124299 (+ 13us) server_negotiation.cc:407] Connection header received
0927 11:16:28.124489 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:28.124492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:28.124513 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:28.124601 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:28.125232 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:28.126195 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:28.127198 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:28.127596 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:28.127709 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:28.129917 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:28.129921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:28.129921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:28.129931 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:28.131773 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:28.132210 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:28.132213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:28.132214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:28.132305 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:28.132568 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:28.132569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:28.132569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:28.132793 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:28.132885 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:28.133168 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 11:16:28.133287 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260927 11:16:28.133407 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:28.124103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43821 (local address 127.0.98.3:40583)
0927 11:16:28.124331 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:28.124341 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:28.124375 (+ 34us) server_negotiation.cc:407] Connection header received
0927 11:16:28.124943 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:28.124946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:28.124983 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:28.125122 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:28.125438 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:28.126480 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:28.127209 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:28.127595 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:28.127722 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:28.130320 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:28.130323 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:28.130323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:28.130333 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:28.131846 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:28.132237 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:28.132239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:28.132239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:28.132315 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:28.132568 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:28.132569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:28.132570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:28.132791 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:28.132881 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:28.133155 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 11:16:28.133275 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":56,"threads_started":1}
I20260927 11:16:28.133415 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:28.124006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40583 (local address 127.0.98.2:43821)
0927 11:16:28.124056 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:28.124080 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:28.124269 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:28.125102 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:28.125103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:28.125279 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:28.125326 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:28.125328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:28.126550 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:28.126551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:28.127113 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:28.127115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:28.127693 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:28.127695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:28.127796 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:28.128409 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:28.128413 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:28.130215 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:28.131935 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:28.131936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:28.131936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:28.132135 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:28.132397 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:28.132398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:28.132399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:28.132496 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:28.132863 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:28.132869 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:28.132994 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 11:16:28.133102 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 11:16:28.133283 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":2}
I20260927 11:16:28.133502 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:28.124025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40657 (local address 127.0.98.2:48315)
0927 11:16:28.124151 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:28.124161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:28.124368 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:28.124651 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:28.124655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:28.124930 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:28.125111 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:28.125116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:28.126291 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:28.126292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:28.127112 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:28.127115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:28.127680 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:28.127681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:28.127798 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:28.128342 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:28.128349 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:28.129803 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:28.131851 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:28.131858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:28.131860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:28.132134 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:28.132366 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:28.132367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:28.132368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:28.132497 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:28.132866 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:28.132872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:28.132994 (+ 122us) client_negotiation.cc:769] Sending connection context
0927 11:16:28.133102 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 11:16:28.133360 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":130,"thread_start_us":61,"threads_started":1}
I20260927 11:16:28.134272 16966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1efaf5ac7c584fe7b598fa5ba85ec60e" candidate_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a9d37f7a1207453b94b44ea2849bbabc" is_pre_election: true
I20260927 11:16:28.134418 16966 raft_consensus.cc:2454] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7beba48c03a4d7fae7e832fa1384bd6 in term 1.
W20260927 11:16:28.134621 17035 leader_election.cc:341] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c3c81a6cd1c49f6a05a952c5f81207f (127.0.98.3:40583): Not found: Tablet not found: 1efaf5ac7c584fe7b598fa5ba85ec60e
I20260927 11:16:28.134693 17035 leader_election.cc:302] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a7beba48c03a4d7fae7e832fa1384bd6, a9d37f7a1207453b94b44ea2849bbabc; no voters: 5c3c81a6cd1c49f6a05a952c5f81207f
I20260927 11:16:28.134800 17291 raft_consensus.cc:2781] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 11:16:28.134856 17291 raft_consensus.cc:482] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:16:28.134884 17291 raft_consensus.cc:3037] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 1 FOLLOWER]: Advancing to term 2
I20260927 11:16:28.135914 17291 raft_consensus.cc:504] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:28.136047 17291 leader_election.cc:288] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [CANDIDATE]: Term 2 election: Requested vote from peers 5c3c81a6cd1c49f6a05a952c5f81207f (127.0.98.3:40583), a9d37f7a1207453b94b44ea2849bbabc (127.0.98.1:40657)
I20260927 11:16:28.136250 16966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1efaf5ac7c584fe7b598fa5ba85ec60e" candidate_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a9d37f7a1207453b94b44ea2849bbabc"
W20260927 11:16:28.136337 17035 leader_election.cc:341] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 5c3c81a6cd1c49f6a05a952c5f81207f (127.0.98.3:40583): Not found: Tablet not found: 1efaf5ac7c584fe7b598fa5ba85ec60e
I20260927 11:16:28.136327 16966 raft_consensus.cc:3037] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc [term 1 FOLLOWER]: Advancing to term 2
I20260927 11:16:28.137445 16966 raft_consensus.cc:2454] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7beba48c03a4d7fae7e832fa1384bd6 in term 2.
I20260927 11:16:28.137586 17037 leader_election.cc:302] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a7beba48c03a4d7fae7e832fa1384bd6, a9d37f7a1207453b94b44ea2849bbabc; no voters: 5c3c81a6cd1c49f6a05a952c5f81207f
I20260927 11:16:28.137673 17291 raft_consensus.cc:2781] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 11:16:28.137779 17291 raft_consensus.cc:686] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [term 2 LEADER]: Becoming Leader. State: Replica: a7beba48c03a4d7fae7e832fa1384bd6, State: Running, Role: LEADER
I20260927 11:16:28.137866 17291 consensus_queue.cc:237] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:28.138105 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:28.137989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:40427)
0927 11:16:28.138027 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:28.138050 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 11:16:28.563019 16966 raft_consensus.cc:1261] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a9d37f7a1207453b94b44ea2849bbabc [term 2 FOLLOWER]: Refusing update from remote peer a7beba48c03a4d7fae7e832fa1384bd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260927 11:16:28.563400 17291 consensus_queue.cc:1035] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260927 11:16:28.564263 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:28.564099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:49955)
0927 11:16:28.564188 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:28.564230 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":54,"threads_started":1}
I20260927 11:16:28.565078 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:28.564984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:53365)
0927 11:16:28.565020 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:28.565050 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260927 11:16:28.565171 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:28.565068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:60267)
0927 11:16:28.565118 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:28.565142 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 11:16:28.565231 17035 consensus_peers.cc:487] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6 -> Peer 5c3c81a6cd1c49f6a05a952c5f81207f (127.0.98.3:40583): Couldn't send request to peer 5c3c81a6cd1c49f6a05a952c5f81207f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1efaf5ac7c584fe7b598fa5ba85ec60e. This is attempt 1: this message will repeat every 5th retry.
I20260927 11:16:28.797931 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:28.797769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:38125)
0927 11:16:28.797869 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:28.797900 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260927 11:16:28.947064 17303 ts_tablet_manager.cc:927] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: Initiating tablet copy from peer a7beba48c03a4d7fae7e832fa1384bd6 (127.0.98.2:40795)
I20260927 11:16:28.947311 17303 tablet_copy_client.cc:316] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:40795
Sep 27 11:16:28 dist-test-slave-9hq9 krb5kdc[15712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 11:16:28.954643 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:28.947558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40795 (local address 127.0.98.3:50907)
0927 11:16:28.947619 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:28.947637 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:28.947712 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:28.947915 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:28.947920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:28.948184 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:28.948255 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:28.948257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:28.949150 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:28.949151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:28.949709 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:28.949710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:28.950253 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:28.950254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:28.950339 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:28.950873 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:28.950878 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:28.952179 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:28.953669 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:28.953674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:28.953676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:28.953853 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:28.954035 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:28.954035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:28.954036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:28.954138 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:28.954390 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:28.954395 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:28.954435 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 11:16:28.954496 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 11:16:28.954572 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 11:16:28.954659 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:28.947677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50907 (local address 127.0.98.2:40795)
0927 11:16:28.947793 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:28.947797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:28.947806 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:16:28.947830 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:28.947832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:28.947843 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:28.947902 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:28.948350 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:28.949077 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:28.949781 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:28.950183 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:28.950240 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:28.952252 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:28.952258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:28.952258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:28.952265 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:28.953600 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:28.953923 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:28.953925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:28.953925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:28.953980 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:28.954194 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:28.954195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:28.954195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:28.954335 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:28.954396 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:28.954533 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 11:16:28.954596 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":51,"threads_started":1}
I20260927 11:16:28.955055 17123 tablet_copy_service.cc:140] P a7beba48c03a4d7fae7e832fa1384bd6: Received BeginTabletCopySession request for tablet 1efaf5ac7c584fe7b598fa5ba85ec60e from peer 5c3c81a6cd1c49f6a05a952c5f81207f ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50907)
I20260927 11:16:28.955113 17123 tablet_copy_service.cc:161] P a7beba48c03a4d7fae7e832fa1384bd6: Beginning new tablet copy session on tablet 1efaf5ac7c584fe7b598fa5ba85ec60e from peer 5c3c81a6cd1c49f6a05a952c5f81207f at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50907: session id = 5c3c81a6cd1c49f6a05a952c5f81207f-1efaf5ac7c584fe7b598fa5ba85ec60e
I20260927 11:16:28.955586 17123 tablet_copy_source_session.cc:205] T 1efaf5ac7c584fe7b598fa5ba85ec60e P a7beba48c03a4d7fae7e832fa1384bd6: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 11:16:28.956480 17303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1efaf5ac7c584fe7b598fa5ba85ec60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:28.959043 17303 tablet_copy_client.cc:791] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: tablet copy: Starting download of 0 data blocks...
I20260927 11:16:28.959195 17303 tablet_copy_client.cc:655] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: tablet copy: Starting download of 1 WAL segments...
I20260927 11:16:28.960014 17303 tablet_copy_client.cc:531] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 11:16:28.960765 17303 tablet_bootstrap.cc:492] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: Bootstrap starting.
I20260927 11:16:28.962037 17303 log.cc:824] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:28.964643 17303 tablet_bootstrap.cc:492] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 11:16:28.964879 17303 tablet_bootstrap.cc:492] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: Bootstrap complete.
I20260927 11:16:28.965026 17303 ts_tablet_manager.cc:1397] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 11:16:28.966040 17303 raft_consensus.cc:348] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:28.966265 17303 raft_consensus.cc:729] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c3c81a6cd1c49f6a05a952c5f81207f, State: Initialized, Role: FOLLOWER
I20260927 11:16:28.966399 17303 consensus_queue.cc:260] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7beba48c03a4d7fae7e832fa1384bd6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40795 } } peers { permanent_uuid: "5c3c81a6cd1c49f6a05a952c5f81207f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40583 } } peers { permanent_uuid: "a9d37f7a1207453b94b44ea2849bbabc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40657 } }
I20260927 11:16:28.966634 17303 ts_tablet_manager.cc:1428] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260927 11:16:28.966902 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:28.966828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:40199)
0927 11:16:28.966853 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:28.966880 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260927 11:16:28.966984 17123 tablet_copy_service.cc:342] P a7beba48c03a4d7fae7e832fa1384bd6: Request end of tablet copy session 5c3c81a6cd1c49f6a05a952c5f81207f-1efaf5ac7c584fe7b598fa5ba85ec60e received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50907
I20260927 11:16:28.967036 17123 tablet_copy_service.cc:433] P a7beba48c03a4d7fae7e832fa1384bd6: ending tablet copy session 5c3c81a6cd1c49f6a05a952c5f81207f-1efaf5ac7c584fe7b598fa5ba85ec60e on tablet 1efaf5ac7c584fe7b598fa5ba85ec60e with peer 5c3c81a6cd1c49f6a05a952c5f81207f
I20260927 11:16:28.967185 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:28.967108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:52885)
0927 11:16:28.967135 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:28.967161 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260927 11:16:29.429136 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:29.428959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:38815)
0927 11:16:29.429040 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:29.429085 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67,"thread_start_us":38,"threads_started":1}
I20260927 11:16:29.429136 17238 raft_consensus.cc:1203] T 1efaf5ac7c584fe7b598fa5ba85ec60e P 5c3c81a6cd1c49f6a05a952c5f81207f [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260927 11:16:29.429574 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:29.429482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:57861)
0927 11:16:29.429517 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:29.429542 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 11:16:29.565671 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:29.565511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:34727)
0927 11:16:29.565603 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:29.565631 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":49,"threads_started":1}
I20260927 11:16:30.429785 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:30.429549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:58199)
0927 11:16:30.429682 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:30.429712 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260927 11:16:30.430254 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:30.430018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:45131)
0927 11:16:30.430180 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:30.430217 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":129,"threads_started":1}
I20260927 11:16:30.566299 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:30.566117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:43677)
0927 11:16:30.566227 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:30.566261 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":53,"threads_started":1}
I20260927 11:16:31.430413 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:31.430218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:41179)
0927 11:16:31.430339 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:31.430369 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260927 11:16:31.430830 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:31.430630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:48679)
0927 11:16:31.430753 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:31.430795 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260927 11:16:31.566934 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:31.566747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:56119)
0927 11:16:31.566859 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:31.566892 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260927 11:16:32.431067 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:32.430860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:40669)
0927 11:16:32.430987 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:32.431022 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260927 11:16:32.431455 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:32.431334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:39733)
0927 11:16:32.431403 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:32.431427 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 11:16:32.431464 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:32.431254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:38865)
0927 11:16:32.431395 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:32.431428 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260927 11:16:32.567581 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:32.567380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:44309)
0927 11:16:32.567504 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:32.567537 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260927 11:16:33.432116 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:33.431888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:42261)
0927 11:16:33.432035 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:33.432070 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":54,"threads_started":1}
I20260927 11:16:33.432116 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:33.431896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:35201)
0927 11:16:33.432044 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:33.432076 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260927 11:16:33.568298 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:33.568101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:55287)
0927 11:16:33.568222 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:33.568257 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260927 11:16:34.432938 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:34.432713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:37969)
0927 11:16:34.432868 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:34.432898 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":73,"threads_started":1}
I20260927 11:16:34.432947 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:34.432711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:36721)
0927 11:16:34.432847 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:34.432885 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260927 11:16:34.568936 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:34.568734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:33271)
0927 11:16:34.568863 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:34.568893 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260927 11:16:35.433636 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:35.433407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:46273)
0927 11:16:35.433543 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:35.433583 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260927 11:16:35.433645 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:35.433417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:43795)
0927 11:16:35.433586 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:35.433610 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
I20260927 11:16:35.569460 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:35.569264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:55949)
0927 11:16:35.569386 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:35.569419 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":53,"threads_started":1}
I20260927 11:16:36.434285 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:36.434103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:36987)
0927 11:16:36.434217 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:36.434241 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
I20260927 11:16:36.434391 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:36.434133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:48069)
0927 11:16:36.434290 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:36.434327 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260927 11:16:36.570164 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:36.569936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:38959)
0927 11:16:36.570083 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:36.570118 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260927 11:16:37.434976 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:37.434814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:50187)
0927 11:16:37.434901 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:37.434933 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":47,"threads_started":1}
I20260927 11:16:37.434998 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:37.434752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:37561)
0927 11:16:37.434875 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:37.434946 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260927 11:16:37.570768 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:37.570591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:47929)
0927 11:16:37.570697 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:37.570728 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260927 11:16:38.435642 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:38.435453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:56339)
0927 11:16:38.435569 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:38.435598 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
I20260927 11:16:38.435662 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:38.435425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:50207)
0927 11:16:38.435580 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:38.435615 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260927 11:16:38.571491 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:38.571232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:46923)
0927 11:16:38.571416 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:38.571450 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":102,"threads_started":1}
I20260927 11:16:39.436424 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:39.436211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:36913)
0927 11:16:39.436351 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:39.436386 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260927 11:16:39.436419 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:39.436182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:41797)
0927 11:16:39.436326 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:39.436361 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260927 11:16:39.572129 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:39.571920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:38373)
0927 11:16:39.572048 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:39.572082 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
I20260927 11:16:40.437122 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:40.436897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:51941)
0927 11:16:40.437052 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:40.437077 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260927 11:16:40.437132 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:40.436900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:50177)
0927 11:16:40.437042 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:40.437082 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260927 11:16:40.572842 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:40.572522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:34475)
0927 11:16:40.572767 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:40.572800 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":71,"threads_started":1}
I20260927 11:16:41.437916 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:41.437733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.2:56603)
0927 11:16:41.437840 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:41.437873 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":51,"threads_started":1}
I20260927 11:16:41.437939 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:41.437718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.3:40391)
0927 11:16:41.437864 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:41.437900 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260927 11:16:41.573556 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:41.573385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36613 (local address 127.0.98.1:51363)
0927 11:16:41.573488 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:41.573519 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
I20260927 11:16:41.808746 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 16850
I20260927 11:16:41.813310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 17018
I20260927 11:16:41.819150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 17154
2026-09-27T11:16:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58396 ms)
[----------] 30 tests from SecurityITest (225803 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:16:41 dist-test-slave-9hq9 krb5kdc[17353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:16:41 dist-test-slave-9hq9 krb5kdc[17353](info): set up 2 sockets
Sep 27 11:16:41 dist-test-slave-9hq9 krb5kdc[17353](info): commencing operation
krb5kdc: starting...
W20260927 11:16:42.853600 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:16:42 dist-test-slave-9hq9 krb5kdc[17353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:16:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:16:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:16:42.961925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37493
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:43.039801 17366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:43.039994 17366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:43.040027 17366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:43.041144 17366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:43.041206 17366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:43.041230 17366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:43.041255 17366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:43.041283 17366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:43.042761 17366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:43.043010 17366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:43.043216 17366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:43.048048 17372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:43.048135 17374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:43.048162 17366 server_base.cc:1034] running on GCE node
W20260927 11:16:43.048147 17371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:43.048661 17366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:43.048930 17366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:43.050104 17366 hybrid_clock.cc:648] HybridClock initialized: now 1790507803050082 us; error 37 us; skew 500 ppm
Sep 27 11:16:43 dist-test-slave-9hq9 krb5kdc[17353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:43.052373 17366 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:16:43.052800 17366 webserver.cc:460] Webserver started at http://127.0.98.62:38747/ using document root <none> and password file <none>
I20260927 11:16:43.053022 17366 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:43.053084 17366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:43.053210 17366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:43.054240 17366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "27bf6c8432934696a6a70b1ff748a10c"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "b33fcee386c937eb5d882dc2cd948166"
server_key_iv: "54c5343ef32afc9fb9ad183a4028b050"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.054603 17366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "27bf6c8432934696a6a70b1ff748a10c"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "b33fcee386c937eb5d882dc2cd948166"
server_key_iv: "54c5343ef32afc9fb9ad183a4028b050"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.055830 17366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:16:43.056504 17381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:43.056668 17366 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:16:43.056761 17366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "27bf6c8432934696a6a70b1ff748a10c"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "b33fcee386c937eb5d882dc2cd948166"
server_key_iv: "54c5343ef32afc9fb9ad183a4028b050"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.056847 17366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:43.081223 17366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:43.082986 17366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:43.083132 17366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:43.087029 17366 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37493
I20260927 11:16:43.087064 17433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37493 every 8 connection(s)
I20260927 11:16:43.087359 17366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:43.088140 17434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:43.090278 17434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c: Bootstrap starting.
I20260927 11:16:43.090802 17434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:43.091074 17434 log.cc:824] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:43.091702 17434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c: No bootstrap required, opened a new log
I20260927 11:16:43.093374 17434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27bf6c8432934696a6a70b1ff748a10c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37493 } }
I20260927 11:16:43.093533 17434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:43.093590 17434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27bf6c8432934696a6a70b1ff748a10c, State: Initialized, Role: FOLLOWER
I20260927 11:16:43.093748 17434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27bf6c8432934696a6a70b1ff748a10c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37493 } }
I20260927 11:16:43.093829 17434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:43.093873 17434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:43.093919 17434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:43.094416 17434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27bf6c8432934696a6a70b1ff748a10c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37493 } }
I20260927 11:16:43.094558 17434 leader_election.cc:302] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27bf6c8432934696a6a70b1ff748a10c; no voters:
I20260927 11:16:43.094738 17434 leader_election.cc:288] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:16:43.094759 17437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:43.094900 17437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [term 1 LEADER]: Becoming Leader. State: Replica: 27bf6c8432934696a6a70b1ff748a10c, State: Running, Role: LEADER
I20260927 11:16:43.094972 17434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:43.095050 17437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27bf6c8432934696a6a70b1ff748a10c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37493 } }
I20260927 11:16:43.095577 17439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27bf6c8432934696a6a70b1ff748a10c. Latest consensus state: current_term: 1 leader_uuid: "27bf6c8432934696a6a70b1ff748a10c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27bf6c8432934696a6a70b1ff748a10c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37493 } } }
I20260927 11:16:43.095669 17439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:43.095897 17438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27bf6c8432934696a6a70b1ff748a10c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27bf6c8432934696a6a70b1ff748a10c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37493 } } }
I20260927 11:16:43.095954 17446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:43.096042 17438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:43.096184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 17366
I20260927 11:16:43.096299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:16:43.096486 17446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:16:43.096503 392 external_mini_cluster.cc:1428] Setting key 9915e4c9ace31dc177a207e8e7beab4c
I20260927 11:16:43.098392 17446 catalog_manager.cc:1347] Generated new cluster ID: d8e68c7665014382b7892d6f67661548
I20260927 11:16:43.098445 17446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:16:43 dist-test-slave-9hq9 krb5kdc[17353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:16:43.107822 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.096872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39548 (local address 127.0.98.62:37493)
0927 11:16:43.097241 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:43.097250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:43.097289 (+ 39us) server_negotiation.cc:407] Connection header received
0927 11:16:43.097370 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:43.097374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:43.097412 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:43.097545 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:43.098360 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.099727 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.100877 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.101443 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.101545 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.104376 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:43.104379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:43.104379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:43.104389 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:43.106288 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:43.106662 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:43.106665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:43.106665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:43.106777 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:43.106975 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:43.106976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:43.106976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:43.107250 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:43.107382 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:43.107491 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 11:16:43.107615 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":65,"threads_started":1}
I20260927 11:16:43.108199 17446 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:16:43.108736 17446 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:43.114720 17446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c: Generated new TSK 0
I20260927 11:16:43.114871 17446 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:16:43.137737 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37493
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:43.216059 17460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:43.216270 17460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:43.216305 17460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:43.217407 17460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:43.217469 17460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:43.217511 17460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:16:43.219058 17460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:43.219305 17460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:43.219513 17460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:43.224232 17466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:43.224234 17465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:43.224232 17468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:43.224501 17460 server_base.cc:1034] running on GCE node
I20260927 11:16:43.224725 17460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:43.224953 17460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:43.226112 17460 hybrid_clock.cc:648] HybridClock initialized: now 1790507803226092 us; error 33 us; skew 500 ppm
Sep 27 11:16:43 dist-test-slave-9hq9 krb5kdc[17353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:43.228449 17460 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:16:43.228889 17460 webserver.cc:460] Webserver started at http://127.0.98.1:42577/ using document root <none> and password file <none>
I20260927 11:16:43.229118 17460 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:43.229163 17460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:43.229305 17460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:43.230347 17460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "c37d12ad1fc34b0bbf24da0443fa93e5"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "469bb000280515c79cdc1015a8aaea86"
server_key_iv: "d4b7d3105ec4f1c8537989e69876c31e"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.230700 17460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c37d12ad1fc34b0bbf24da0443fa93e5"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "469bb000280515c79cdc1015a8aaea86"
server_key_iv: "d4b7d3105ec4f1c8537989e69876c31e"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.231966 17460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:16:43.232686 17475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:43.232851 17460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:43.232942 17460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "c37d12ad1fc34b0bbf24da0443fa93e5"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "469bb000280515c79cdc1015a8aaea86"
server_key_iv: "d4b7d3105ec4f1c8537989e69876c31e"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.233022 17460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:43.245692 17460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:43.247433 17460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:43.247579 17460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:43.247792 17460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:43.248068 17460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:43.248123 17460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:43.248162 17460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:43.248186 17460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:43.253973 17460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36655
I20260927 11:16:43.254021 17588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36655 every 8 connection(s)
I20260927 11:16:43.254315 17460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:16:43 dist-test-slave-9hq9 krb5kdc[17353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:16:43.262224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 17460
I20260927 11:16:43.262346 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:16:43.262537 392 external_mini_cluster.cc:1428] Setting key 6cb19a2a022f3fedb6f63a3f8280c0ac
I20260927 11:16:43.263031 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.254961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38331 (local address 127.0.98.62:37493)
0927 11:16:43.255005 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:43.255007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:43.255200 (+ 193us) server_negotiation.cc:407] Connection header received
0927 11:16:43.256047 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:43.256047 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:43.256059 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:43.256129 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:43.256676 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.257160 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.258136 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.258332 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.258383 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.260394 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:43.260396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:43.260396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:43.260402 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:43.261546 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:43.261953 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:43.261954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:43.261954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:43.262026 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:43.262313 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:43.262314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:43.262314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:43.262455 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:43.262524 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:43.262851 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 11:16:43.262939 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:16:43.263101 17591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.254934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37493 (local address 127.0.98.1:38331)
0927 11:16:43.255077 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:43.255115 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:43.255965 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:43.256183 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:43.256186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:43.256418 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:43.256591 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.256595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.257236 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.257237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.258005 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.258007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.258401 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.258402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.258502 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.258915 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:43.258923 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:43.260267 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:43.261640 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:43.261647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:43.261649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:43.261865 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:43.262135 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:43.262136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:43.262136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:43.262239 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:43.262526 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:43.262533 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:43.262659 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 11:16:43.262798 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 11:16:43.262938 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260927 11:16:43.263584 17589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37493
I20260927 11:16:43.263667 17589 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:43.263839 17589 heartbeater.cc:511] Master 127.0.98.62:37493 requested a full tablet report, sending...
I20260927 11:16:43.264370 17398 ts_manager.cc:194] Registered new tserver with Master: c37d12ad1fc34b0bbf24da0443fa93e5 (127.0.98.1:36655)
I20260927 11:16:43.265079 17398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38331
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:16:43.284746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37493
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:43.363276 17594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:43.363452 17594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:43.363473 17594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:43.364573 17594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:43.364641 17594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:43.364696 17594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:16:43.366179 17594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:43.366344 17594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:43.366557 17594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:43.371203 17600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:43.371337 17599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:43.371338 17602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:43.371236 17594 server_base.cc:1034] running on GCE node
I20260927 11:16:43.371610 17594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:43.371831 17594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:43.372990 17594 hybrid_clock.cc:648] HybridClock initialized: now 1790507803372987 us; error 35 us; skew 500 ppm
Sep 27 11:16:43 dist-test-slave-9hq9 krb5kdc[17353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:43.375352 17594 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:16:43.375684 17594 webserver.cc:460] Webserver started at http://127.0.98.2:37183/ using document root <none> and password file <none>
I20260927 11:16:43.375901 17594 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:43.375944 17594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:43.376097 17594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:43.377044 17594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "19fb399a75374f14b98da97f9c57686b"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "be2cd9895b2d1181348619dc538b9a26"
server_key_iv: "51e2809a1ba00c0194a988ee43d15aee"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.377388 17594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "19fb399a75374f14b98da97f9c57686b"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "be2cd9895b2d1181348619dc538b9a26"
server_key_iv: "51e2809a1ba00c0194a988ee43d15aee"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.378700 17594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:16:43.379365 17609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:43.379547 17594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:43.379611 17594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "19fb399a75374f14b98da97f9c57686b"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "be2cd9895b2d1181348619dc538b9a26"
server_key_iv: "51e2809a1ba00c0194a988ee43d15aee"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.379695 17594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:43.388897 17594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:43.390805 17594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:43.390945 17594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:43.391160 17594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:43.391438 17594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:43.391489 17594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:43.391530 17594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:43.391558 17594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:43.397152 17594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42749
I20260927 11:16:43.397240 17722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42749 every 8 connection(s)
I20260927 11:16:43.397486 17594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:16:43.398907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 17594
I20260927 11:16:43.399024 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:16:43.399215 392 external_mini_cluster.cc:1428] Setting key 9406f3a371073bab1eac33f679a1b00c
Sep 27 11:16:43 dist-test-slave-9hq9 krb5kdc[17353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:16:43.406972 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.398027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37493 (local address 127.0.98.2:48527)
0927 11:16:43.398199 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:43.398228 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:43.399130 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:43.399431 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:43.399435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:43.399717 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:43.399938 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.399943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.400597 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.400598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.401780 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.401783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.402139 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.402140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.402278 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.402676 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:43.402685 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:43.404033 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:43.405743 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:43.405748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:43.405750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:43.405953 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:43.406192 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:43.406192 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:43.406193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:43.406289 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:43.406560 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:43.406565 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:43.406660 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 11:16:43.406735 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 11:16:43.406839 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":111,"threads_started":1}
I20260927 11:16:43.406993 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.398101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48527 (local address 127.0.98.62:37493)
0927 11:16:43.398153 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:43.398156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:43.398323 (+ 167us) server_negotiation.cc:407] Connection header received
0927 11:16:43.399224 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:43.399225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:43.399235 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:43.399313 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:43.400022 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.400513 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.401864 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.402036 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.402134 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.404114 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:43.404115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:43.404116 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:43.404123 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:43.405656 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:43.406023 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:43.406025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:43.406025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:43.406106 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:43.406355 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:43.406356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:43.406356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:43.406492 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:43.406565 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:43.406785 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 11:16:43.406877 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 11:16:43.407581 17723 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37493
I20260927 11:16:43.407670 17723 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 11:16:43.407835 17723 heartbeater.cc:511] Master 127.0.98.62:37493 requested a full tablet report, sending...
I20260927 11:16:43.408205 17398 ts_manager.cc:194] Registered new tserver with Master: 19fb399a75374f14b98da97f9c57686b (127.0.98.2:42749)
I20260927 11:16:43.408571 17398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:16:43.422765 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37493
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:43.500171 17728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:43.500349 17728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:43.500370 17728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:43.501353 17728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:43.501391 17728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:43.501415 17728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:16:43.502641 17728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:43.502789 17728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:43.502928 17728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:43.507525 17734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:43.507464 17733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:43.507464 17736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:43.507767 17728 server_base.cc:1034] running on GCE node
I20260927 11:16:43.507969 17728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:43.508184 17728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:43.509347 17728 hybrid_clock.cc:648] HybridClock initialized: now 1790507803509344 us; error 34 us; skew 500 ppm
Sep 27 11:16:43 dist-test-slave-9hq9 krb5kdc[17353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:43.511704 17728 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:16:43.512099 17728 webserver.cc:460] Webserver started at http://127.0.98.3:36233/ using document root <none> and password file <none>
I20260927 11:16:43.512302 17728 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:43.512377 17728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:43.512523 17728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:43.513463 17728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "6f532442465f4ecd8474d150ebf71920"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "5aa577b31b871b697729b5168a95a97b"
server_key_iv: "748559bef7d08c6ba815071c5eb9f9ee"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.513799 17728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6f532442465f4ecd8474d150ebf71920"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "5aa577b31b871b697729b5168a95a97b"
server_key_iv: "748559bef7d08c6ba815071c5eb9f9ee"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.515106 17728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:16:43.515769 17743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:43.515939 17728 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:16:43.516042 17728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "6f532442465f4ecd8474d150ebf71920"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "5aa577b31b871b697729b5168a95a97b"
server_key_iv: "748559bef7d08c6ba815071c5eb9f9ee"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.516151 17728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:43.540656 17728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:43.542850 17728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:43.543017 17728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:43.543252 17728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:43.543541 17728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:43.543593 17728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:43.543632 17728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:43.543661 17728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:43.549068 17728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44881
I20260927 11:16:43.549108 17856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44881 every 8 connection(s)
I20260927 11:16:43.549423 17728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 11:16:43 dist-test-slave-9hq9 krb5kdc[17353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:16:43.556970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 17728
I20260927 11:16:43.557080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:16:43.557292 392 external_mini_cluster.cc:1428] Setting key 708f5d9931ad31435d039f3ca0bf8351
I20260927 11:16:43.557963 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.550132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47145 (local address 127.0.98.62:37493)
0927 11:16:43.550185 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:43.550187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:43.550362 (+ 175us) server_negotiation.cc:407] Connection header received
0927 11:16:43.551214 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:43.551215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:43.551224 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:43.551278 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:43.551772 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.552242 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.553145 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.553316 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.553370 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.555219 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:43.555220 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:43.555220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:43.555226 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:43.556596 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:43.556931 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:43.556933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:43.556933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:43.556996 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:43.557247 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:43.557248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:43.557248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:43.557399 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:43.557471 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:43.557769 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 11:16:43.557846 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 11:16:43.558224 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.550091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37493 (local address 127.0.98.3:47145)
0927 11:16:43.550254 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:43.550292 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:43.551131 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:43.551324 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:43.551327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:43.551544 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:43.551707 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.551711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.552313 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.552314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.553075 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.553076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.553387 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.553388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.553483 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.553871 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:43.553878 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:43.555140 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:43.556661 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:43.556665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:43.556667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:43.556858 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:43.557074 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:43.557075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:43.557076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:43.557197 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:43.557471 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:43.557477 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:43.557617 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 11:16:43.557735 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 11:16:43.557906 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260927 11:16:43.558596 17857 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37493
I20260927 11:16:43.558697 17857 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:43.558871 17857 heartbeater.cc:511] Master 127.0.98.62:37493 requested a full tablet report, sending...
I20260927 11:16:43.559242 17398 ts_manager.cc:194] Registered new tserver with Master: 6f532442465f4ecd8474d150ebf71920 (127.0.98.3:44881)
I20260927 11:16:43.559640 17398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47145
I20260927 11:16:43.569082 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:16:43.576184 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.570458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39552 (local address 127.0.98.62:37493)
0927 11:16:43.570537 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:43.570539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:43.570622 (+ 83us) server_negotiation.cc:407] Connection header received
0927 11:16:43.570714 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:43.570715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:43.570724 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:43.570777 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:43.571353 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.571803 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.572537 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.572760 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.572840 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.573902 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:43.573903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:43.573904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:43.573910 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:43.575030 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:43.575368 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:43.575369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:43.575369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:43.575420 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:43.575626 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:43.575627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:43.575628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:43.575766 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:43.575867 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:43.576015 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 11:16:43.576087 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 11:16:43.577696 17398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:16:43.579286 17398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:16:43.585347 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.581575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44881 (local address 127.0.0.1:47298)
0927 11:16:43.581680 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:43.581689 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:43.581735 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:43.582341 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:43.582342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:43.582344 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:43.582386 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.582388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.583524 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.583526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.584398 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.584401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.585043 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.585045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.585122 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.585134 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 11:16:43.585224 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 11:16:43.585274 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260927 11:16:43.585351 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.581665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.0.98.3:44881)
0927 11:16:43.582201 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:43.582205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:43.582214 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:16:43.582247 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:43.582248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:43.582264 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:43.582330 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:43.582456 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.583450 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.584479 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.584945 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.585039 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.585094 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:43.585239 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 11:16:43.585274 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":57,"threads_started":1}
I20260927 11:16:43.585619 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.581459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36655 (local address 127.0.0.1:37162)
0927 11:16:43.581675 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:43.581684 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:43.581735 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:43.582153 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:43.582154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:43.582157 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:43.582200 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.582202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.583472 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.583473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.584458 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.584460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.585384 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.585385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.585464 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.585469 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:16:43.585504 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 11:16:43.585543 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":9,"thread_start_us":38,"threads_started":1}
I20260927 11:16:43.585809 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.581237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42749 (local address 127.0.0.1:36366)
0927 11:16:43.581345 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:43.581374 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:43.581426 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:43.581979 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:43.581983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:43.581985 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:43.582041 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.582043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.583077 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.583079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.584437 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.584440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.585534 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.585535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.585585 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.585588 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:43.585628 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 11:16:43.585751 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":73,"threads_started":1}
I20260927 11:16:43.585963 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.581591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37162 (local address 127.0.98.1:36655)
0927 11:16:43.581854 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:43.581858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:43.581865 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:16:43.581894 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:43.581896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:43.581907 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:43.581995 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:43.582313 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.583375 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.584539 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.585270 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.585737 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.585802 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:43.585828 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 11:16:43.585877 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":106,"threads_started":1}
I20260927 11:16:43.586004 17791 tablet_service.cc:1467] Processing CreateTablet for tablet b0dbc8c8792148c288dd8d46a64e8c60 (DEFAULT_TABLE table=test-table [id=ee8fe1645a7c4d488d3986d8f49ea869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:43.586228 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.581350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36366 (local address 127.0.98.2:42749)
0927 11:16:43.581829 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:43.581833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:43.581841 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:16:43.581873 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:43.581874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:43.581885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:43.581974 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:43.582153 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.582975 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.584538 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.585270 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.585341 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.585959 (+ 618us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:43.585998 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 11:16:43.586127 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":115,"threads_started":1}
I20260927 11:16:43.586381 17791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0dbc8c8792148c288dd8d46a64e8c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:43.586570 17523 tablet_service.cc:1467] Processing CreateTablet for tablet b0dbc8c8792148c288dd8d46a64e8c60 (DEFAULT_TABLE table=test-table [id=ee8fe1645a7c4d488d3986d8f49ea869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:43.586774 17657 tablet_service.cc:1467] Processing CreateTablet for tablet b0dbc8c8792148c288dd8d46a64e8c60 (DEFAULT_TABLE table=test-table [id=ee8fe1645a7c4d488d3986d8f49ea869]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:43.586853 17523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0dbc8c8792148c288dd8d46a64e8c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:43.586993 17657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0dbc8c8792148c288dd8d46a64e8c60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:43.589566 17872 tablet_bootstrap.cc:492] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920: Bootstrap starting.
I20260927 11:16:43.589895 17873 tablet_bootstrap.cc:492] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5: Bootstrap starting.
I20260927 11:16:43.590196 17871 tablet_bootstrap.cc:492] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b: Bootstrap starting.
I20260927 11:16:43.590221 17872 tablet_bootstrap.cc:654] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:43.590510 17873 tablet_bootstrap.cc:654] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:43.590811 17872 log.cc:824] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:43.590847 17871 tablet_bootstrap.cc:654] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:43.590993 17873 log.cc:824] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:43.591452 17871 log.cc:824] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:43.591583 17872 tablet_bootstrap.cc:492] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920: No bootstrap required, opened a new log
I20260927 11:16:43.591656 17872 ts_tablet_manager.cc:1397] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 11:16:43.591787 17873 tablet_bootstrap.cc:492] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5: No bootstrap required, opened a new log
I20260927 11:16:43.591848 17873 ts_tablet_manager.cc:1397] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:16:43.593596 17872 raft_consensus.cc:348] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb399a75374f14b98da97f9c57686b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36655 } } peers { permanent_uuid: "6f532442465f4ecd8474d150ebf71920" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44881 } }
I20260927 11:16:43.593765 17872 raft_consensus.cc:374] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:43.593802 17872 raft_consensus.cc:729] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f532442465f4ecd8474d150ebf71920, State: Initialized, Role: FOLLOWER
I20260927 11:16:43.593808 17873 raft_consensus.cc:348] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb399a75374f14b98da97f9c57686b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36655 } } peers { permanent_uuid: "6f532442465f4ecd8474d150ebf71920" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44881 } }
I20260927 11:16:43.593961 17873 raft_consensus.cc:374] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:43.593942 17872 consensus_queue.cc:260] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb399a75374f14b98da97f9c57686b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36655 } } peers { permanent_uuid: "6f532442465f4ecd8474d150ebf71920" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44881 } }
I20260927 11:16:43.594004 17873 raft_consensus.cc:729] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37d12ad1fc34b0bbf24da0443fa93e5, State: Initialized, Role: FOLLOWER
I20260927 11:16:43.594115 17873 consensus_queue.cc:260] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb399a75374f14b98da97f9c57686b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36655 } } peers { permanent_uuid: "6f532442465f4ecd8474d150ebf71920" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44881 } }
I20260927 11:16:43.594224 17872 ts_tablet_manager.cc:1428] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 11:16:43.594255 17857 heartbeater.cc:503] Master 127.0.98.62:37493 was elected leader, sending a full tablet report...
I20260927 11:16:43.594363 17873 ts_tablet_manager.cc:1428] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:16:43.594491 17589 heartbeater.cc:503] Master 127.0.98.62:37493 was elected leader, sending a full tablet report...
I20260927 11:16:43.595224 17871 tablet_bootstrap.cc:492] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b: No bootstrap required, opened a new log
I20260927 11:16:43.595297 17871 ts_tablet_manager.cc:1397] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260927 11:16:43.596931 17871 raft_consensus.cc:348] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb399a75374f14b98da97f9c57686b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36655 } } peers { permanent_uuid: "6f532442465f4ecd8474d150ebf71920" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44881 } }
I20260927 11:16:43.597078 17871 raft_consensus.cc:374] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:43.597100 17871 raft_consensus.cc:729] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19fb399a75374f14b98da97f9c57686b, State: Initialized, Role: FOLLOWER
I20260927 11:16:43.597215 17871 consensus_queue.cc:260] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb399a75374f14b98da97f9c57686b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36655 } } peers { permanent_uuid: "6f532442465f4ecd8474d150ebf71920" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44881 } }
I20260927 11:16:43.597440 17871 ts_tablet_manager.cc:1428] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:16:43.597489 17723 heartbeater.cc:503] Master 127.0.98.62:37493 was elected leader, sending a full tablet report...
I20260927 11:16:43.644620 17878 raft_consensus.cc:482] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:16:43.644769 17878 raft_consensus.cc:504] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb399a75374f14b98da97f9c57686b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36655 } } peers { permanent_uuid: "6f532442465f4ecd8474d150ebf71920" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44881 } }
I20260927 11:16:43.645112 17878 leader_election.cc:288] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19fb399a75374f14b98da97f9c57686b (127.0.98.2:42749), 6f532442465f4ecd8474d150ebf71920 (127.0.98.3:44881)
I20260927 11:16:43.646509 17879 raft_consensus.cc:482] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:16:43.646600 17879 raft_consensus.cc:504] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb399a75374f14b98da97f9c57686b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36655 } } peers { permanent_uuid: "6f532442465f4ecd8474d150ebf71920" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44881 } }
I20260927 11:16:43.646919 17879 leader_election.cc:288] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c37d12ad1fc34b0bbf24da0443fa93e5 (127.0.98.1:36655), 6f532442465f4ecd8474d150ebf71920 (127.0.98.3:44881)
W20260927 11:16:43.648315 17724 tablet.cc:2369] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:16:43.648638 17591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.645250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42749 (local address 127.0.98.1:43943)
0927 11:16:43.645285 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:43.645297 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:43.645338 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:43.645569 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:43.645570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:43.645573 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:43.645642 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.645644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.646420 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.646421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.647658 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.647660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.648401 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.648402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.648489 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.648492 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:43.648535 (+ 43us) client_negotiation.cc:240] Negotiation successful
0927 11:16:43.648570 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15,"mutex_wait_us":84}
I20260927 11:16:43.648674 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.645429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56975 (local address 127.0.98.3:44881)
0927 11:16:43.645484 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:43.645486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:43.645574 (+ 88us) server_negotiation.cc:407] Connection header received
0927 11:16:43.645619 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:43.645620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:43.645629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:43.645679 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:43.645839 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.646284 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.647766 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.648302 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.648342 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.648352 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:43.648575 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 11:16:43.648615 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:16:43.648880 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.645354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44881 (local address 127.0.98.1:56975)
0927 11:16:43.645500 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:43.645511 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:43.645566 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:43.645704 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:43.645705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:43.645707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:43.645773 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.645774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.646349 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.646350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.647677 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.647679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.648421 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.648421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.648493 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.648497 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:16:43.648765 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 11:16:43.648807 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":110,"thread_start_us":84,"threads_started":1}
I20260927 11:16:43.649139 17811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dbc8c8792148c288dd8d46a64e8c60" candidate_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f532442465f4ecd8474d150ebf71920" is_pre_election: true
I20260927 11:16:43.649195 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.645365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43943 (local address 127.0.98.2:42749)
0927 11:16:43.645409 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:43.645411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:43.645419 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:16:43.645445 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:43.645446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:43.645458 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:43.645533 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:43.645717 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.646333 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.647742 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.648303 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.648362 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.648371 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:43.649106 (+ 735us) server_negotiation.cc:299] Negotiation successful
0927 11:16:43.649140 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:16:43.649268 17811 raft_consensus.cc:2454] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37d12ad1fc34b0bbf24da0443fa93e5 in term 0.
I20260927 11:16:43.649484 17476 leader_election.cc:302] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f532442465f4ecd8474d150ebf71920, c37d12ad1fc34b0bbf24da0443fa93e5; no voters:
I20260927 11:16:43.649600 17878 raft_consensus.cc:2781] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:16:43.649648 17878 raft_consensus.cc:482] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:16:43.649682 17878 raft_consensus.cc:3037] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:43.649842 17677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dbc8c8792148c288dd8d46a64e8c60" candidate_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb399a75374f14b98da97f9c57686b" is_pre_election: true
I20260927 11:16:43.649928 17677 raft_consensus.cc:2454] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c37d12ad1fc34b0bbf24da0443fa93e5 in term 0.
I20260927 11:16:43.650533 17878 raft_consensus.cc:504] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb399a75374f14b98da97f9c57686b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36655 } } peers { permanent_uuid: "6f532442465f4ecd8474d150ebf71920" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44881 } }
I20260927 11:16:43.650635 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.647049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36655 (local address 127.0.98.2:39507)
0927 11:16:43.647225 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:43.647233 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:43.647275 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:43.647421 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:43.647422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:43.647424 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:43.647471 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.647473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.648525 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.648526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.649606 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.649609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.650411 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.650412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.650490 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.650493 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:43.650544 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 11:16:43.650576 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260927 11:16:43.650642 17878 leader_election.cc:288] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [CANDIDATE]: Term 1 election: Requested vote from peers 19fb399a75374f14b98da97f9c57686b (127.0.98.2:42749), 6f532442465f4ecd8474d150ebf71920 (127.0.98.3:44881)
I20260927 11:16:43.650825 17811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dbc8c8792148c288dd8d46a64e8c60" candidate_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f532442465f4ecd8474d150ebf71920"
I20260927 11:16:43.650826 17677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dbc8c8792148c288dd8d46a64e8c60" candidate_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19fb399a75374f14b98da97f9c57686b"
I20260927 11:16:43.650894 17677 raft_consensus.cc:3037] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:43.650887 17811 raft_consensus.cc:3037] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:43.651134 17881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.647129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44881 (local address 127.0.98.2:34631)
0927 11:16:43.647786 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:43.647795 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:43.647842 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:43.648017 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:43.648018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:43.648019 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:43.648077 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.648078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.648884 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.648884 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.650131 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.650133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.650885 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.650886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.650947 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.650950 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:43.651026 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 11:16:43.651065 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":65,"threads_started":1}
I20260927 11:16:43.651175 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.647373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34631 (local address 127.0.98.3:44881)
0927 11:16:43.647544 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:43.647545 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:43.647889 (+ 344us) server_negotiation.cc:407] Connection header received
0927 11:16:43.647911 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:43.647911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:43.647919 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:43.647983 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:43.648137 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.648798 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.650217 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.650803 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.650867 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.650878 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:43.651065 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 11:16:43.651101 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260927 11:16:43.651647 17811 raft_consensus.cc:2454] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37d12ad1fc34b0bbf24da0443fa93e5 in term 1.
I20260927 11:16:43.651647 17677 raft_consensus.cc:2454] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c37d12ad1fc34b0bbf24da0443fa93e5 in term 1.
I20260927 11:16:43.651829 17476 leader_election.cc:302] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19fb399a75374f14b98da97f9c57686b, c37d12ad1fc34b0bbf24da0443fa93e5; no voters:
I20260927 11:16:43.651880 17811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dbc8c8792148c288dd8d46a64e8c60" candidate_uuid: "19fb399a75374f14b98da97f9c57686b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f532442465f4ecd8474d150ebf71920" is_pre_election: true
I20260927 11:16:43.651995 17811 raft_consensus.cc:2379] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 19fb399a75374f14b98da97f9c57686b in current term 1: Already voted for candidate c37d12ad1fc34b0bbf24da0443fa93e5 in this term.
I20260927 11:16:43.651997 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.647081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39507 (local address 127.0.98.1:36655)
0927 11:16:43.647124 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:43.647126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:43.647284 (+ 158us) server_negotiation.cc:407] Connection header received
0927 11:16:43.647323 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:43.647324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:43.647332 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:43.647385 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:43.647532 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.648172 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.649786 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.650322 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.650398 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.650407 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:43.651883 (+ 1476us) server_negotiation.cc:299] Negotiation successful
0927 11:16:43.651930 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 11:16:43.652086 17878 raft_consensus.cc:2781] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:43.652192 17878 raft_consensus.cc:686] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [term 1 LEADER]: Becoming Leader. State: Replica: c37d12ad1fc34b0bbf24da0443fa93e5, State: Running, Role: LEADER
I20260927 11:16:43.652225 17543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dbc8c8792148c288dd8d46a64e8c60" candidate_uuid: "19fb399a75374f14b98da97f9c57686b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" is_pre_election: true
I20260927 11:16:43.652268 17878 consensus_queue.cc:237] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb399a75374f14b98da97f9c57686b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36655 } } peers { permanent_uuid: "6f532442465f4ecd8474d150ebf71920" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44881 } }
I20260927 11:16:43.652801 17610 leader_election.cc:302] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 19fb399a75374f14b98da97f9c57686b; no voters: 6f532442465f4ecd8474d150ebf71920, c37d12ad1fc34b0bbf24da0443fa93e5
I20260927 11:16:43.652915 17879 raft_consensus.cc:2726] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 11:16:43.652980 17398 catalog_manager.cc:5495] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c37d12ad1fc34b0bbf24da0443fa93e5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19fb399a75374f14b98da97f9c57686b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c37d12ad1fc34b0bbf24da0443fa93e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f532442465f4ecd8474d150ebf71920" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44881 } health_report { overall_health: UNKNOWN } } }
I20260927 11:16:43.675568 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.673484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.0.98.1:36655)
0927 11:16:43.673527 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:43.673529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:43.673538 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:16:43.673599 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:43.673600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:43.673615 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:43.673675 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:16:43.673784 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.674297 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.674979 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.675147 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.675196 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.675313 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:16:43.675401 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:16:43.675453 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:43.675487 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 11:16:43.675500 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:16:43.677212 17811 raft_consensus.cc:1261] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920 [term 1 FOLLOWER]: Refusing update from remote peer c37d12ad1fc34b0bbf24da0443fa93e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:16:43.677229 17677 raft_consensus.cc:1261] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b [term 1 FOLLOWER]: Refusing update from remote peer c37d12ad1fc34b0bbf24da0443fa93e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:16:43.677503 17878 consensus_queue.cc:1035] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19fb399a75374f14b98da97f9c57686b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:16:43.677611 17883 consensus_queue.cc:1035] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f532442465f4ecd8474d150ebf71920" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:16:43.680047 17886 mvcc.cc:205] Tried to move back new op lower bound from 7333919963858378752 to 7333919963759460352. Current Snapshot: MvccSnapshot[applied={T|T < 7333919963858378752}]
I20260927 11:16:43.680364 17887 mvcc.cc:205] Tried to move back new op lower bound from 7333919963858378752 to 7333919963759460352. Current Snapshot: MvccSnapshot[applied={T|T < 7333919963858378752}]
I20260927 11:16:43.682591 17398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:16:43.682686 17398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:16:43.683385 17398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27bf6c8432934696a6a70b1ff748a10c: Sending DeleteTablet for 3 replicas of tablet b0dbc8c8792148c288dd8d46a64e8c60
I20260927 11:16:43.683753 17523 tablet_service.cc:1514] Processing DeleteTablet for tablet b0dbc8c8792148c288dd8d46a64e8c60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37162
I20260927 11:16:43.683925 17657 tablet_service.cc:1514] Processing DeleteTablet for tablet b0dbc8c8792148c288dd8d46a64e8c60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36366
I20260927 11:16:43.683972 17896 tablet_replica.cc:330] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5: stopping tablet replica
I20260927 11:16:43.684162 17896 raft_consensus.cc:2229] T b0dbc8c8792148c288dd8d46a64e8c60 P c37d12ad1fc34b0bbf24da0443fa93e5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:16:43.684142 17791 tablet_service.cc:1514] Processing DeleteTablet for tablet b0dbc8c8792148c288dd8d46a64e8c60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47298
I20260927 11:16:43.684204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 17460
I20260927 11:16:43.685036 17898 tablet_replica.cc:330] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920: stopping tablet replica
I20260927 11:16:43.685108 17898 raft_consensus.cc:2229] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:16:43.685173 17898 raft_consensus.cc:2258] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:43.685523 17898 ts_tablet_manager.cc:1905] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:16:43.686308 17897 tablet_replica.cc:330] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b: stopping tablet replica
I20260927 11:16:43.686407 17897 raft_consensus.cc:2229] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:16:43.686501 17897 raft_consensus.cc:2258] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:43.686884 17897 ts_tablet_manager.cc:1905] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:16:43.687034 17898 ts_tablet_manager.cc:1918] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:16:43.687107 17898 log.cc:1196] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/b0dbc8c8792148c288dd8d46a64e8c60
I20260927 11:16:43.687371 17898 ts_tablet_manager.cc:1939] T b0dbc8c8792148c288dd8d46a64e8c60 P 6f532442465f4ecd8474d150ebf71920: Deleting consensus metadata
I20260927 11:16:43.687758 17382 catalog_manager.cc:4841] TS 6f532442465f4ecd8474d150ebf71920 (127.0.98.3:44881): tablet b0dbc8c8792148c288dd8d46a64e8c60 (table test-table [id=ee8fe1645a7c4d488d3986d8f49ea869]) successfully deleted
I20260927 11:16:43.688441 17897 ts_tablet_manager.cc:1918] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:16:43.688496 17897 log.cc:1196] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/b0dbc8c8792148c288dd8d46a64e8c60
I20260927 11:16:43.688758 17897 ts_tablet_manager.cc:1939] T b0dbc8c8792148c288dd8d46a64e8c60 P 19fb399a75374f14b98da97f9c57686b: Deleting consensus metadata
I20260927 11:16:43.689150 17382 catalog_manager.cc:4841] TS 19fb399a75374f14b98da97f9c57686b (127.0.98.2:42749): tablet b0dbc8c8792148c288dd8d46a64e8c60 (table test-table [id=ee8fe1645a7c4d488d3986d8f49ea869]) successfully deleted
W20260927 11:16:43.689906 17382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36655 (error 108)
I20260927 11:16:43.690379 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.690276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36655 (local address 127.0.0.1:37184)
0927 11:16:43.690325 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:43.690355 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 11:16:43.690480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 17594
W20260927 11:16:43.690512 17382 catalog_manager.cc:4584] TS c37d12ad1fc34b0bbf24da0443fa93e5 (127.0.98.1:36655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0dbc8c8792148c288dd8d46a64e8c60: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36655: connect: Connection refused (error 111)
I20260927 11:16:43.695765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 17728
I20260927 11:16:43.700179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 17366
2026-09-27T11:16:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1881 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T11:16:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:16:43Z Disabled control of system clock
I20260927 11:16:43.728350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39955
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:16:43.805361 17903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:43.805593 17903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:43.805625 17903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:43.807056 17903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:43.807117 17903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:43.807140 17903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:43.807161 17903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:43.807178 17903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:43.809504 17903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:43.809767 17903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:43.810133 17903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:43.815210 17911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:43.815267 17909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:43.815222 17908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:43.815388 17903 server_base.cc:1034] running on GCE node
I20260927 11:16:43.815629 17903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:43.815920 17903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:43.817080 17903 hybrid_clock.cc:648] HybridClock initialized: now 1790507803817049 us; error 48 us; skew 500 ppm
I20260927 11:16:43.818293 17903 webserver.cc:460] Webserver started at http://127.0.98.62:38537/ using document root <none> and password file <none>
I20260927 11:16:43.818532 17903 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:43.818608 17903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:43.818744 17903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:43.819696 17903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "5027d21e59144f119a024fb473785e14"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "10cc276290d9d877030fe9668e5af54b"
server_key_iv: "105f754dc4416639506f5b11afe438ba"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.820029 17903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "5027d21e59144f119a024fb473785e14"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "10cc276290d9d877030fe9668e5af54b"
server_key_iv: "105f754dc4416639506f5b11afe438ba"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.821321 17903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:16:43.822021 17917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:43.822208 17903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:43.822302 17903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "5027d21e59144f119a024fb473785e14"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "10cc276290d9d877030fe9668e5af54b"
server_key_iv: "105f754dc4416639506f5b11afe438ba"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.822381 17903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:43.830964 17903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:43.831192 17903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:43.831323 17903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:43.834962 17903 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39955
I20260927 11:16:43.834988 17969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39955 every 8 connection(s)
I20260927 11:16:43.835278 17903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:43.836123 17970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:43.838014 17970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14: Bootstrap starting.
I20260927 11:16:43.838548 17970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:43.838770 17970 log.cc:824] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:43.839375 17970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14: No bootstrap required, opened a new log
I20260927 11:16:43.841285 17970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027d21e59144f119a024fb473785e14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } }
I20260927 11:16:43.841459 17970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:43.841496 17970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5027d21e59144f119a024fb473785e14, State: Initialized, Role: FOLLOWER
I20260927 11:16:43.841617 17970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027d21e59144f119a024fb473785e14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } }
I20260927 11:16:43.841684 17970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:43.841710 17970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:43.841768 17970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:43.842401 17970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027d21e59144f119a024fb473785e14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } }
I20260927 11:16:43.842449 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 17903
I20260927 11:16:43.842507 17970 leader_election.cc:302] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5027d21e59144f119a024fb473785e14; no voters:
I20260927 11:16:43.842618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:16:43.842648 17970 leader_election.cc:288] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:16:43.842805 17974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:43.842872 392 external_mini_cluster.cc:1428] Setting key 3ae60d48baf3f25d2925c34ca470df61
I20260927 11:16:43.842876 17970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:43.843027 17974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [term 1 LEADER]: Becoming Leader. State: Replica: 5027d21e59144f119a024fb473785e14, State: Running, Role: LEADER
I20260927 11:16:43.843163 17974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027d21e59144f119a024fb473785e14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } }
I20260927 11:16:43.843552 17973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5027d21e59144f119a024fb473785e14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027d21e59144f119a024fb473785e14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } }
I20260927 11:16:43.843643 17973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:43.843557 17974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5027d21e59144f119a024fb473785e14. Latest consensus state: current_term: 1 leader_uuid: "5027d21e59144f119a024fb473785e14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027d21e59144f119a024fb473785e14" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39955 } } }
I20260927 11:16:43.843816 17974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14 [sys.catalog]: This master's current role is: LEADER
W20260927 11:16:43.844951 17990 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:16:43.845007 17990 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:16:43.845049 17984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:43.845184 17984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:16:43.846983 17984 catalog_manager.cc:1347] Generated new cluster ID: 3d4c3c1b10c54ffe9c5e30f6685001cd
I20260927 11:16:43.847034 17984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:16:43.848872 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.843351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34166 (local address 127.0.98.62:39955)
0927 11:16:43.843895 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:43.843920 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:43.843957 (+ 37us) server_negotiation.cc:407] Connection header received
0927 11:16:43.844070 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:43.844076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:43.844140 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:43.844263 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:43.844499 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.845844 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.846935 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.847483 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.847598 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.848241 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:43.848244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:43.848244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:43.848419 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:43.848453 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:43.848465 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:43.848521 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:43.848623 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 11:16:43.848686 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260927 11:16:43.858847 17984 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:16:43.859294 17984 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:43.862982 17984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14: Generated new TSK 0
I20260927 11:16:43.863121 17984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:16:43.866994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39955
--builtin_ntp_servers=127.0.98.20:40407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:16:43.945604 17994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:43.945786 17994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:43.945807 17994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:43.947000 17994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:43.947041 17994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:43.947067 17994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:16:43.948364 17994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:43.948585 17994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:43.948906 17994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:43.953919 17999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:43.953938 18000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:43.954089 17994 server_base.cc:1034] running on GCE node
W20260927 11:16:43.954313 18002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:43.954577 17994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:43.954818 17994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:43.955962 17994 hybrid_clock.cc:648] HybridClock initialized: now 1790507803955949 us; error 33 us; skew 500 ppm
I20260927 11:16:43.957158 17994 webserver.cc:460] Webserver started at http://127.0.98.1:37649/ using document root <none> and password file <none>
I20260927 11:16:43.957388 17994 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:43.957456 17994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:43.957605 17994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:43.958667 17994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "f9b63bc8fd5343e49aa7e73b19e80261"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "b9902b591af1f4f54ed55956c39dd69d"
server_key_iv: "09b2acbbd4f1beaba109196cc68689b3"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.959010 17994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f9b63bc8fd5343e49aa7e73b19e80261"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "b9902b591af1f4f54ed55956c39dd69d"
server_key_iv: "09b2acbbd4f1beaba109196cc68689b3"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.960290 17994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:16:43.961051 18008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:43.961203 17994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:43.961282 17994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "f9b63bc8fd5343e49aa7e73b19e80261"
format_stamp: "Formatted at 2026-09-27 11:16:43 on dist-test-slave-9hq9"
server_key: "b9902b591af1f4f54ed55956c39dd69d"
server_key_iv: "09b2acbbd4f1beaba109196cc68689b3"
server_key_version: "encryptionkey@0"
I20260927 11:16:43.961366 17994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:43.976454 17994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:43.976723 17994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:43.976866 17994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:43.977080 17994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:43.977375 17994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:43.977432 17994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:43.977473 17994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:43.977514 17994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:43.982919 17994 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42149
I20260927 11:16:43.982964 18121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42149 every 8 connection(s)
I20260927 11:16:43.983248 17994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:16:43.988147 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.983877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34409 (local address 127.0.98.62:39955)
0927 11:16:43.983944 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:43.983946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:43.984148 (+ 202us) server_negotiation.cc:407] Connection header received
0927 11:16:43.984647 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:43.984648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:43.984660 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:43.984722 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:43.985348 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.985835 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.986797 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.987003 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.987091 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.987881 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:43.987883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:43.987883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:43.987887 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:43.987900 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:43.987904 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:43.987949 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:43.988043 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 11:16:43.988074 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:16:43.988344 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:43.983850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39955 (local address 127.0.98.1:34409)
0927 11:16:43.984038 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:43.984069 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:43.984538 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:43.984791 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:43.984794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:43.984802 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:43.985260 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.985263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.985940 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.985942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.986705 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:43.986707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:43.987100 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:43.987101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:43.987222 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:43.987755 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:16:43.987755 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:43.987779 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:16:43.987798 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:43.987948 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:43.987953 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:43.987955 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:16:43.988031 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 11:16:43.988120 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260927 11:16:43.988860 18122 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39955
I20260927 11:16:43.988947 18122 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:43.989070 18122 heartbeater.cc:511] Master 127.0.98.62:39955 requested a full tablet report, sending...
I20260927 11:16:43.989590 17934 ts_manager.cc:194] Registered new tserver with Master: f9b63bc8fd5343e49aa7e73b19e80261 (127.0.98.1:42149)
I20260927 11:16:43.990190 17934 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:34409
I20260927 11:16:43.991380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 17994
I20260927 11:16:43.991447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:16:43.991657 392 external_mini_cluster.cc:1428] Setting key 93ba017330dbdedf64ff737ce9b7fcb7
I20260927 11:16:43.992498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39955
--builtin_ntp_servers=127.0.98.20:40407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:16:44.068365 18125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:44.068585 18125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:44.068619 18125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:44.069733 18125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:44.069794 18125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:44.069835 18125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:16:44.071171 18125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:44.071398 18125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:44.071645 18125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:44.076092 18133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:44.076128 18131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:44.076100 18130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:44.076231 18125 server_base.cc:1034] running on GCE node
I20260927 11:16:44.076551 18125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:44.076789 18125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:44.077943 18125 hybrid_clock.cc:648] HybridClock initialized: now 1790507804077926 us; error 27 us; skew 500 ppm
I20260927 11:16:44.079114 18125 webserver.cc:460] Webserver started at http://127.0.98.2:46463/ using document root <none> and password file <none>
I20260927 11:16:44.079336 18125 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:44.079399 18125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:44.079543 18125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:44.080542 18125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "b4041ab04aa341b689293a5613dc5dc9"
format_stamp: "Formatted at 2026-09-27 11:16:44 on dist-test-slave-9hq9"
server_key: "b1e42c0ca51ff81f61a1ee56e03898df"
server_key_iv: "de1c1587eb4e7d765a8d112b1fd2bcd9"
server_key_version: "encryptionkey@0"
I20260927 11:16:44.080874 18125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b4041ab04aa341b689293a5613dc5dc9"
format_stamp: "Formatted at 2026-09-27 11:16:44 on dist-test-slave-9hq9"
server_key: "b1e42c0ca51ff81f61a1ee56e03898df"
server_key_iv: "de1c1587eb4e7d765a8d112b1fd2bcd9"
server_key_version: "encryptionkey@0"
I20260927 11:16:44.082075 18125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:16:44.082772 18139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:44.082922 18125 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:44.083010 18125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "b4041ab04aa341b689293a5613dc5dc9"
format_stamp: "Formatted at 2026-09-27 11:16:44 on dist-test-slave-9hq9"
server_key: "b1e42c0ca51ff81f61a1ee56e03898df"
server_key_iv: "de1c1587eb4e7d765a8d112b1fd2bcd9"
server_key_version: "encryptionkey@0"
I20260927 11:16:44.083091 18125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:44.094276 18125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:44.094574 18125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:44.094718 18125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:44.094928 18125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:44.095220 18125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:44.095276 18125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:44.095314 18125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:44.095341 18125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:44.100864 18125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34889
I20260927 11:16:44.100914 18252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34889 every 8 connection(s)
I20260927 11:16:44.101186 18125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:16:44.105651 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.101801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43227 (local address 127.0.98.62:39955)
0927 11:16:44.101863 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:44.101865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:44.102098 (+ 233us) server_negotiation.cc:407] Connection header received
0927 11:16:44.102533 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:44.102534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:44.102543 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:44.102597 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:44.103196 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.103685 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.104561 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.104742 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.104791 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.105437 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:44.105438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:44.105438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:44.105442 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:44.105455 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:44.105459 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:44.105501 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:44.105577 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 11:16:44.105598 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:16:44.105810 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.101790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39955 (local address 127.0.98.2:43227)
0927 11:16:44.101964 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:44.102001 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:44.102437 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:44.102669 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:44.102671 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:44.102679 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:44.103129 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:44.103133 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.103769 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.103770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:44.104494 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:44.104496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.104810 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.104811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:44.104910 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.105322 (+ 412us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:16:44.105322 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:44.105345 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:16:44.105363 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:44.105512 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:44.105515 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:44.105517 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:16:44.105566 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 11:16:44.105639 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260927 11:16:44.106285 18253 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39955
I20260927 11:16:44.106374 18253 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:44.106526 18253 heartbeater.cc:511] Master 127.0.98.62:39955 requested a full tablet report, sending...
I20260927 11:16:44.106778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 18125
I20260927 11:16:44.106901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:16:44.106921 17934 ts_manager.cc:194] Registered new tserver with Master: b4041ab04aa341b689293a5613dc5dc9 (127.0.98.2:34889)
I20260927 11:16:44.107117 392 external_mini_cluster.cc:1428] Setting key 9bce06268f35d2354b8bc47cca12b2f5
I20260927 11:16:44.107402 17934 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:43227
I20260927 11:16:44.108194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39955
--builtin_ntp_servers=127.0.98.20:40407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:16:44.184837 18256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:44.185038 18256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:44.185066 18256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:44.186234 18256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:44.186283 18256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:44.186308 18256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:16:44.187572 18256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:44.187775 18256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:44.188011 18256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:44.192615 18262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:44.192638 18261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:44.192615 18264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:44.192770 18256 server_base.cc:1034] running on GCE node
I20260927 11:16:44.193007 18256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:44.193243 18256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:44.194375 18256 hybrid_clock.cc:648] HybridClock initialized: now 1790507804194360 us; error 32 us; skew 500 ppm
I20260927 11:16:44.195515 18256 webserver.cc:460] Webserver started at http://127.0.98.3:45357/ using document root <none> and password file <none>
I20260927 11:16:44.195737 18256 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:44.195780 18256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:44.195914 18256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:44.196882 18256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "5e08920ade09462b8a30a544a9a99a80"
format_stamp: "Formatted at 2026-09-27 11:16:44 on dist-test-slave-9hq9"
server_key: "6d228e0dd2fdb0ab3472b16c023b159b"
server_key_iv: "019447bb419fb7354fd0621998e533cc"
server_key_version: "encryptionkey@0"
I20260927 11:16:44.197219 18256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5e08920ade09462b8a30a544a9a99a80"
format_stamp: "Formatted at 2026-09-27 11:16:44 on dist-test-slave-9hq9"
server_key: "6d228e0dd2fdb0ab3472b16c023b159b"
server_key_iv: "019447bb419fb7354fd0621998e533cc"
server_key_version: "encryptionkey@0"
I20260927 11:16:44.198503 18256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:16:44.199150 18270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:44.199314 18256 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:44.199406 18256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "5e08920ade09462b8a30a544a9a99a80"
format_stamp: "Formatted at 2026-09-27 11:16:44 on dist-test-slave-9hq9"
server_key: "6d228e0dd2fdb0ab3472b16c023b159b"
server_key_iv: "019447bb419fb7354fd0621998e533cc"
server_key_version: "encryptionkey@0"
I20260927 11:16:44.199481 18256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:44.228312 18256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:44.228543 18256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:44.228672 18256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:44.228890 18256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:44.229184 18256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:44.229235 18256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:44.229269 18256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:44.229295 18256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:44.234814 18256 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36179
I20260927 11:16:44.234860 18383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36179 every 8 connection(s)
I20260927 11:16:44.235139 18256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:16:44.239336 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.235824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50317 (local address 127.0.98.62:39955)
0927 11:16:44.235890 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:44.235892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:44.236064 (+ 172us) server_negotiation.cc:407] Connection header received
0927 11:16:44.236477 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:44.236478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:44.236487 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:44.236542 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:44.237107 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.237562 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.238383 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.238545 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.238593 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.239167 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:44.239168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:44.239168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:44.239172 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:44.239183 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:44.239186 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:44.239226 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:44.239268 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 11:16:44.239288 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:16:44.239488 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.235793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39955 (local address 127.0.98.3:50317)
0927 11:16:44.235949 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:44.235985 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:44.236404 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:44.236587 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:44.236590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:44.236597 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:44.237036 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:44.237039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.237631 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.237631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:44.238316 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:44.238318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.238610 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.238610 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:44.238699 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.239064 (+ 365us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:16:44.239064 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:44.239085 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:16:44.239102 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:44.239228 (+ 126us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:44.239230 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:44.239232 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:16:44.239269 (+ 37us) client_negotiation.cc:240] Negotiation successful
0927 11:16:44.239328 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260927 11:16:44.239929 18384 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39955
I20260927 11:16:44.240114 18384 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:44.240274 18384 heartbeater.cc:511] Master 127.0.98.62:39955 requested a full tablet report, sending...
I20260927 11:16:44.240620 17933 ts_manager.cc:194] Registered new tserver with Master: 5e08920ade09462b8a30a544a9a99a80 (127.0.98.3:36179)
I20260927 11:16:44.240949 17933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50317
I20260927 11:16:44.242946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 18256
I20260927 11:16:44.243057 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:16:44.243239 392 external_mini_cluster.cc:1428] Setting key 4708a427f8d79a811e589b4628113fb1
I20260927 11:16:44.244063 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:16:44.247737 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.245345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.0.98.62:39955)
0927 11:16:44.245381 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:44.245383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:44.245481 (+ 98us) server_negotiation.cc:407] Connection header received
0927 11:16:44.245539 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:44.245540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:44.245549 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:44.245596 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:44.245727 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.246214 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.246862 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.247033 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.247079 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.247571 (+ 492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:44.247572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:44.247572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:44.247576 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:44.247585 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:44.247589 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:44.247629 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:44.247676 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 11:16:44.247692 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:16:44.249179 17933 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:16:44.250752 17933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:16:44.256621 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.252858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42149 (local address 127.0.0.1:55320)
0927 11:16:44.253135 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:44.253146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:44.253188 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:44.253418 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:44.253418 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:44.253420 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:44.253459 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:44.253461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.254395 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.254395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:44.255532 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:44.255534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.256353 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.256354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:44.256422 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.256433 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 11:16:44.256499 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 11:16:44.256546 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":65,"threads_started":1}
I20260927 11:16:44.256630 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.252956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.0.98.1:42149)
0927 11:16:44.253163 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:44.253166 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:44.253195 (+ 29us) server_negotiation.cc:407] Connection header received
0927 11:16:44.253265 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:44.253269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:44.253283 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:44.253355 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:44.253534 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.254316 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.255600 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.256269 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.256337 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.256413 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:44.256508 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 11:16:44.256543 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":108,"threads_started":1}
I20260927 11:16:44.256971 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.252637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34889 (local address 127.0.0.1:35952)
0927 11:16:44.252971 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:44.252980 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:44.253017 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:44.253227 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:44.253230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:44.253232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:44.253285 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:44.253287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.254766 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.254767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:44.255624 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:44.255626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.256476 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.256477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:44.256528 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.256531 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:44.256575 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 11:16:44.256603 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":4,"thread_start_us":72,"threads_started":1}
I20260927 11:16:44.257066 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.252729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.0.98.2:34889)
0927 11:16:44.253005 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:44.253010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:44.253023 (+ 13us) server_negotiation.cc:407] Connection header received
0927 11:16:44.253095 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:44.253098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:44.253115 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:44.253214 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:44.253422 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.254665 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.255703 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.256283 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.256363 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.256444 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:44.256661 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 11:16:44.256957 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260927 11:16:44.257352 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.252636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36179 (local address 127.0.0.1:42902)
0927 11:16:44.252772 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:44.252810 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:44.252870 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:44.253481 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:44.253482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:44.253484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:44.253530 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:44.253531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.254757 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.254758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:44.255839 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:44.255841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.256828 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.256829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:44.256872 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.256875 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:44.256909 (+ 34us) client_negotiation.cc:240] Negotiation successful
0927 11:16:44.256966 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":24,"thread_start_us":77,"threads_started":1}
I20260927 11:16:44.257387 18056 tablet_service.cc:1467] Processing CreateTablet for tablet 070b0787b2ed4de4ba2f8c4fbbeba438 (DEFAULT_TABLE table=test-table [id=643b6468bd134d4e84fb8c954becc4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:44.257652 18187 tablet_service.cc:1467] Processing CreateTablet for tablet 070b0787b2ed4de4ba2f8c4fbbeba438 (DEFAULT_TABLE table=test-table [id=643b6468bd134d4e84fb8c954becc4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:44.257764 18056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070b0787b2ed4de4ba2f8c4fbbeba438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:44.257787 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.252980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42902 (local address 127.0.98.3:36179)
0927 11:16:44.253298 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:44.253304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:44.253317 (+ 13us) server_negotiation.cc:407] Connection header received
0927 11:16:44.253365 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:44.253367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:44.253379 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:44.253602 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:44.253642 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.254671 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.255918 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.256580 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.256645 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.257650 (+ 1005us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:44.257676 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 11:16:44.257715 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":84,"threads_started":1}
I20260927 11:16:44.257975 18187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070b0787b2ed4de4ba2f8c4fbbeba438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:44.258345 18318 tablet_service.cc:1467] Processing CreateTablet for tablet 070b0787b2ed4de4ba2f8c4fbbeba438 (DEFAULT_TABLE table=test-table [id=643b6468bd134d4e84fb8c954becc4ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:44.258638 18318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070b0787b2ed4de4ba2f8c4fbbeba438. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:44.260268 18398 tablet_bootstrap.cc:492] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261: Bootstrap starting.
I20260927 11:16:44.260905 18398 tablet_bootstrap.cc:654] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:44.261200 18399 tablet_bootstrap.cc:492] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9: Bootstrap starting.
I20260927 11:16:44.261260 18398 log.cc:824] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:44.261792 18399 tablet_bootstrap.cc:654] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:44.261973 18398 tablet_bootstrap.cc:492] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261: No bootstrap required, opened a new log
I20260927 11:16:44.262030 18398 ts_tablet_manager.cc:1397] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:16:44.262154 18399 log.cc:824] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:44.262207 18401 tablet_bootstrap.cc:492] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80: Bootstrap starting.
I20260927 11:16:44.262755 18401 tablet_bootstrap.cc:654] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:44.262873 18399 tablet_bootstrap.cc:492] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9: No bootstrap required, opened a new log
I20260927 11:16:44.262940 18399 ts_tablet_manager.cc:1397] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:16:44.263058 18401 log.cc:824] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:44.264286 18398 raft_consensus.cc:348] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e08920ade09462b8a30a544a9a99a80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36179 } } peers { permanent_uuid: "b4041ab04aa341b689293a5613dc5dc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34889 } } peers { permanent_uuid: "f9b63bc8fd5343e49aa7e73b19e80261" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42149 } }
I20260927 11:16:44.264426 18398 raft_consensus.cc:374] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:44.264468 18398 raft_consensus.cc:729] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b63bc8fd5343e49aa7e73b19e80261, State: Initialized, Role: FOLLOWER
I20260927 11:16:44.264568 18398 consensus_queue.cc:260] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e08920ade09462b8a30a544a9a99a80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36179 } } peers { permanent_uuid: "b4041ab04aa341b689293a5613dc5dc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34889 } } peers { permanent_uuid: "f9b63bc8fd5343e49aa7e73b19e80261" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42149 } }
I20260927 11:16:44.264837 18398 ts_tablet_manager.cc:1428] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:16:44.264914 18122 heartbeater.cc:503] Master 127.0.98.62:39955 was elected leader, sending a full tablet report...
I20260927 11:16:44.265028 18399 raft_consensus.cc:348] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e08920ade09462b8a30a544a9a99a80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36179 } } peers { permanent_uuid: "b4041ab04aa341b689293a5613dc5dc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34889 } } peers { permanent_uuid: "f9b63bc8fd5343e49aa7e73b19e80261" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42149 } }
I20260927 11:16:44.265141 18399 raft_consensus.cc:374] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:44.265201 18399 raft_consensus.cc:729] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4041ab04aa341b689293a5613dc5dc9, State: Initialized, Role: FOLLOWER
I20260927 11:16:44.265316 18399 consensus_queue.cc:260] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e08920ade09462b8a30a544a9a99a80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36179 } } peers { permanent_uuid: "b4041ab04aa341b689293a5613dc5dc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34889 } } peers { permanent_uuid: "f9b63bc8fd5343e49aa7e73b19e80261" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42149 } }
I20260927 11:16:44.265550 18399 ts_tablet_manager.cc:1428] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:16:44.265582 18253 heartbeater.cc:503] Master 127.0.98.62:39955 was elected leader, sending a full tablet report...
I20260927 11:16:44.266289 18401 tablet_bootstrap.cc:492] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80: No bootstrap required, opened a new log
I20260927 11:16:44.266382 18401 ts_tablet_manager.cc:1397] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 11:16:44.268244 18401 raft_consensus.cc:348] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e08920ade09462b8a30a544a9a99a80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36179 } } peers { permanent_uuid: "b4041ab04aa341b689293a5613dc5dc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34889 } } peers { permanent_uuid: "f9b63bc8fd5343e49aa7e73b19e80261" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42149 } }
I20260927 11:16:44.268381 18401 raft_consensus.cc:374] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:44.268426 18401 raft_consensus.cc:729] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e08920ade09462b8a30a544a9a99a80, State: Initialized, Role: FOLLOWER
I20260927 11:16:44.268528 18401 consensus_queue.cc:260] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e08920ade09462b8a30a544a9a99a80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36179 } } peers { permanent_uuid: "b4041ab04aa341b689293a5613dc5dc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34889 } } peers { permanent_uuid: "f9b63bc8fd5343e49aa7e73b19e80261" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42149 } }
I20260927 11:16:44.268762 18401 ts_tablet_manager.cc:1428] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 11:16:44.268819 18384 heartbeater.cc:503] Master 127.0.98.62:39955 was elected leader, sending a full tablet report...
W20260927 11:16:44.351698 18254 tablet.cc:2369] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:16:44.360894 18405 raft_consensus.cc:482] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:16:44.360980 18405 raft_consensus.cc:504] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e08920ade09462b8a30a544a9a99a80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36179 } } peers { permanent_uuid: "b4041ab04aa341b689293a5613dc5dc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34889 } } peers { permanent_uuid: "f9b63bc8fd5343e49aa7e73b19e80261" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42149 } }
I20260927 11:16:44.361254 18405 leader_election.cc:288] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e08920ade09462b8a30a544a9a99a80 (127.0.98.3:36179), f9b63bc8fd5343e49aa7e73b19e80261 (127.0.98.1:42149)
I20260927 11:16:44.364064 18255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.361418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36179 (local address 127.0.98.2:45701)
0927 11:16:44.361457 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:44.361476 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:44.361560 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:44.361703 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:44.361704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:44.361706 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:44.361780 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:44.361782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.362489 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.362490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:44.363203 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:44.363205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.363808 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.363809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:44.363881 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.363884 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:44.363956 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 11:16:44.363999 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 11:16:44.364063 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.361488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45701 (local address 127.0.98.3:36179)
0927 11:16:44.361567 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:44.361569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:44.361593 (+ 24us) server_negotiation.cc:407] Connection header received
0927 11:16:44.361616 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:44.361617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:44.361624 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:44.361683 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:44.361858 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.362387 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.363304 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.363718 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.363811 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.363820 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:44.363963 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 11:16:44.364000 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 11:16:44.364395 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.361488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48121 (local address 127.0.98.1:42149)
0927 11:16:44.361573 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:44.361575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:44.361839 (+ 264us) server_negotiation.cc:407] Connection header received
0927 11:16:44.361904 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:44.361905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:44.361916 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:44.361967 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:44.362160 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.362633 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.363660 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.364019 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.364098 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.364109 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:44.364285 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 11:16:44.364326 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 11:16:44.364398 18338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070b0787b2ed4de4ba2f8c4fbbeba438" candidate_uuid: "b4041ab04aa341b689293a5613dc5dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e08920ade09462b8a30a544a9a99a80" is_pre_election: true
I20260927 11:16:44.364557 18338 raft_consensus.cc:2454] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4041ab04aa341b689293a5613dc5dc9 in term 0.
I20260927 11:16:44.364655 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.361418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42149 (local address 127.0.98.2:48121)
0927 11:16:44.361765 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:44.361774 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:44.361850 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:44.361978 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:44.361979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:44.361980 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:44.362102 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:44.362105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.362716 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.362717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:44.363551 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:44.363553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.364119 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.364120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:44.364185 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.364187 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:16:44.364503 (+ 316us) client_negotiation.cc:240] Negotiation successful
0927 11:16:44.364548 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":198,"thread_start_us":97,"threads_started":1}
I20260927 11:16:44.364862 18141 leader_election.cc:302] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e08920ade09462b8a30a544a9a99a80, b4041ab04aa341b689293a5613dc5dc9; no voters:
I20260927 11:16:44.364991 18405 raft_consensus.cc:2781] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:16:44.365077 18405 raft_consensus.cc:482] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:16:44.365118 18405 raft_consensus.cc:3037] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:44.365252 18076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070b0787b2ed4de4ba2f8c4fbbeba438" candidate_uuid: "b4041ab04aa341b689293a5613dc5dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b63bc8fd5343e49aa7e73b19e80261" is_pre_election: true
I20260927 11:16:44.365387 18076 raft_consensus.cc:2454] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4041ab04aa341b689293a5613dc5dc9 in term 0.
I20260927 11:16:44.365898 18405 raft_consensus.cc:504] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e08920ade09462b8a30a544a9a99a80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36179 } } peers { permanent_uuid: "b4041ab04aa341b689293a5613dc5dc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34889 } } peers { permanent_uuid: "f9b63bc8fd5343e49aa7e73b19e80261" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42149 } }
I20260927 11:16:44.366037 18405 leader_election.cc:288] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [CANDIDATE]: Term 1 election: Requested vote from peers 5e08920ade09462b8a30a544a9a99a80 (127.0.98.3:36179), f9b63bc8fd5343e49aa7e73b19e80261 (127.0.98.1:42149)
I20260927 11:16:44.366197 18338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070b0787b2ed4de4ba2f8c4fbbeba438" candidate_uuid: "b4041ab04aa341b689293a5613dc5dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e08920ade09462b8a30a544a9a99a80"
I20260927 11:16:44.366225 18076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070b0787b2ed4de4ba2f8c4fbbeba438" candidate_uuid: "b4041ab04aa341b689293a5613dc5dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b63bc8fd5343e49aa7e73b19e80261"
I20260927 11:16:44.366273 18338 raft_consensus.cc:3037] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:44.366283 18076 raft_consensus.cc:3037] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:44.367118 18076 raft_consensus.cc:2454] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4041ab04aa341b689293a5613dc5dc9 in term 1.
I20260927 11:16:44.367118 18338 raft_consensus.cc:2454] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4041ab04aa341b689293a5613dc5dc9 in term 1.
I20260927 11:16:44.367270 18141 leader_election.cc:302] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e08920ade09462b8a30a544a9a99a80, b4041ab04aa341b689293a5613dc5dc9; no voters:
I20260927 11:16:44.367420 18405 raft_consensus.cc:2781] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:44.367564 18405 raft_consensus.cc:686] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [term 1 LEADER]: Becoming Leader. State: Replica: b4041ab04aa341b689293a5613dc5dc9, State: Running, Role: LEADER
I20260927 11:16:44.367678 18405 consensus_queue.cc:237] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e08920ade09462b8a30a544a9a99a80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36179 } } peers { permanent_uuid: "b4041ab04aa341b689293a5613dc5dc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34889 } } peers { permanent_uuid: "f9b63bc8fd5343e49aa7e73b19e80261" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42149 } }
I20260927 11:16:44.368446 17934 catalog_manager.cc:5495] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4041ab04aa341b689293a5613dc5dc9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b4041ab04aa341b689293a5613dc5dc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e08920ade09462b8a30a544a9a99a80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4041ab04aa341b689293a5613dc5dc9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9b63bc8fd5343e49aa7e73b19e80261" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42149 } health_report { overall_health: UNKNOWN } } }
I20260927 11:16:44.372965 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.370801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.0.98.2:34889)
0927 11:16:44.370860 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:44.370862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:44.370871 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:16:44.370923 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:44.370923 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:44.370938 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:44.371005 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:16:44.371102 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.371522 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.372243 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:44.372444 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:44.372508 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:44.372650 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:16:44.372735 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:16:44.372844 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:44.372862 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 11:16:44.372878 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 11:16:44.374660 18338 raft_consensus.cc:1261] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80 [term 1 FOLLOWER]: Refusing update from remote peer b4041ab04aa341b689293a5613dc5dc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:16:44.374660 18076 raft_consensus.cc:1261] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261 [term 1 FOLLOWER]: Refusing update from remote peer b4041ab04aa341b689293a5613dc5dc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:16:44.374995 18408 consensus_queue.cc:1035] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9b63bc8fd5343e49aa7e73b19e80261" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:16:44.375118 18405 consensus_queue.cc:1035] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e08920ade09462b8a30a544a9a99a80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:16:44.377136 18411 mvcc.cc:205] Tried to move back new op lower bound from 7333919966714982400 to 7333919966690066432. Current Snapshot: MvccSnapshot[applied={T|T < 7333919966714982400}]
I20260927 11:16:44.377760 18412 mvcc.cc:205] Tried to move back new op lower bound from 7333919966714982400 to 7333919966690066432. Current Snapshot: MvccSnapshot[applied={T|T < 7333919966714982400}]
I20260927 11:16:44.377763 18413 mvcc.cc:205] Tried to move back new op lower bound from 7333919966714982400 to 7333919966690066432. Current Snapshot: MvccSnapshot[applied={T|T < 7333919966714982400}]
I20260927 11:16:44.379191 17934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:16:44.379272 17934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:16:44.379976 17934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5027d21e59144f119a024fb473785e14: Sending DeleteTablet for 3 replicas of tablet 070b0787b2ed4de4ba2f8c4fbbeba438
I20260927 11:16:44.380290 18318 tablet_service.cc:1514] Processing DeleteTablet for tablet 070b0787b2ed4de4ba2f8c4fbbeba438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:44 UTC) from {username='slave'} at 127.0.0.1:42902
I20260927 11:16:44.380318 18187 tablet_service.cc:1514] Processing DeleteTablet for tablet 070b0787b2ed4de4ba2f8c4fbbeba438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:44 UTC) from {username='slave'} at 127.0.0.1:35952
I20260927 11:16:44.380355 18056 tablet_service.cc:1514] Processing DeleteTablet for tablet 070b0787b2ed4de4ba2f8c4fbbeba438 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:44 UTC) from {username='slave'} at 127.0.0.1:55320
I20260927 11:16:44.380606 18422 tablet_replica.cc:330] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261: stopping tablet replica
I20260927 11:16:44.380704 18422 raft_consensus.cc:2229] T 070b0787b2ed4de4ba2f8c4fbbeba438 P f9b63bc8fd5343e49aa7e73b19e80261 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:16:44.380764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 17994
I20260927 11:16:44.381384 18421 tablet_replica.cc:330] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9: stopping tablet replica
I20260927 11:16:44.381479 18421 raft_consensus.cc:2229] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:16:44.381567 18421 raft_consensus.cc:2258] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:44.381959 18421 ts_tablet_manager.cc:1905] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:16:44.383797 18421 ts_tablet_manager.cc:1918] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:16:44.383872 18421 log.cc:1196] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/070b0787b2ed4de4ba2f8c4fbbeba438
I20260927 11:16:44.384122 18421 ts_tablet_manager.cc:1939] T 070b0787b2ed4de4ba2f8c4fbbeba438 P b4041ab04aa341b689293a5613dc5dc9: Deleting consensus metadata
I20260927 11:16:44.384871 17921 catalog_manager.cc:4841] TS b4041ab04aa341b689293a5613dc5dc9 (127.0.98.2:34889): tablet 070b0787b2ed4de4ba2f8c4fbbeba438 (table test-table [id=643b6468bd134d4e84fb8c954becc4ec]) successfully deleted
I20260927 11:16:44.386152 18420 tablet_replica.cc:330] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80: stopping tablet replica
I20260927 11:16:44.386253 18420 raft_consensus.cc:2229] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:16:44.386363 18420 raft_consensus.cc:2258] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:44.386796 18420 ts_tablet_manager.cc:1905] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 11:16:44.388012 17921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42149 (error 108)
I20260927 11:16:44.388234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 18125
I20260927 11:16:44.388309 18420 ts_tablet_manager.cc:1918] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:16:44.388386 18420 log.cc:1196] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/070b0787b2ed4de4ba2f8c4fbbeba438
I20260927 11:16:44.388617 18420 ts_tablet_manager.cc:1939] T 070b0787b2ed4de4ba2f8c4fbbeba438 P 5e08920ade09462b8a30a544a9a99a80: Deleting consensus metadata
I20260927 11:16:44.388978 17919 catalog_manager.cc:4841] TS 5e08920ade09462b8a30a544a9a99a80 (127.0.98.3:36179): tablet 070b0787b2ed4de4ba2f8c4fbbeba438 (table test-table [id=643b6468bd134d4e84fb8c954becc4ec]) successfully deleted
I20260927 11:16:44.390902 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:44.390757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42149 (local address 127.0.0.1:55330)
0927 11:16:44.390829 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:44.390881 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 11:16:44.391000 17921 catalog_manager.cc:4584] TS f9b63bc8fd5343e49aa7e73b19e80261 (127.0.98.1:42149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 070b0787b2ed4de4ba2f8c4fbbeba438: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42149: connect: Connection refused (error 111)
I20260927 11:16:44.393630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 18256
I20260927 11:16:44.398244 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 17903
2026-09-27T11:16:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (696 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:16:44 dist-test-slave-9hq9 krb5kdc[18426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:16:44 dist-test-slave-9hq9 krb5kdc[18426](info): set up 2 sockets
Sep 27 11:16:44 dist-test-slave-9hq9 krb5kdc[18426](info): commencing operation
krb5kdc: starting...
W20260927 11:16:45.413834 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:16:45 dist-test-slave-9hq9 krb5kdc[18426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:16:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:16:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:16:45.523002 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45393
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:45.598487 18439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:45.598681 18439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:45.598702 18439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:45.599817 18439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:45.599856 18439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:45.599870 18439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:45.599915 18439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:45.599943 18439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:45.601336 18439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:45.601576 18439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:45.601796 18439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:45.606258 18445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:45.606268 18444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:45.606283 18439 server_base.cc:1034] running on GCE node
W20260927 11:16:45.606241 18447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:45.606695 18439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:45.606913 18439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:45.608039 18439 hybrid_clock.cc:648] HybridClock initialized: now 1790507805608014 us; error 42 us; skew 500 ppm
Sep 27 11:16:45 dist-test-slave-9hq9 krb5kdc[18426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:45.610265 18439 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:16:45.610599 18439 webserver.cc:460] Webserver started at http://127.0.98.62:40805/ using document root <none> and password file <none>
I20260927 11:16:45.610764 18439 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:45.610801 18439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:45.610890 18439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:45.611774 18439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "29bd2a32ca30475f9e89b93b2b94dc79"
format_stamp: "Formatted at 2026-09-27 11:16:45 on dist-test-slave-9hq9"
server_key: "b389a72a7e17a824f7377288efb90e93"
server_key_iv: "e41a3b844053bba9a8aea60279e719fc"
server_key_version: "encryptionkey@0"
I20260927 11:16:45.612063 18439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "29bd2a32ca30475f9e89b93b2b94dc79"
format_stamp: "Formatted at 2026-09-27 11:16:45 on dist-test-slave-9hq9"
server_key: "b389a72a7e17a824f7377288efb90e93"
server_key_iv: "e41a3b844053bba9a8aea60279e719fc"
server_key_version: "encryptionkey@0"
I20260927 11:16:45.613214 18439 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:16:45.613824 18454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:45.613994 18439 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:45.614110 18439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "29bd2a32ca30475f9e89b93b2b94dc79"
format_stamp: "Formatted at 2026-09-27 11:16:45 on dist-test-slave-9hq9"
server_key: "b389a72a7e17a824f7377288efb90e93"
server_key_iv: "e41a3b844053bba9a8aea60279e719fc"
server_key_version: "encryptionkey@0"
I20260927 11:16:45.614173 18439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:45.640719 18439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:45.642388 18439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:45.642489 18439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:45.646037 18439 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45393
I20260927 11:16:45.646124 18506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45393 every 8 connection(s)
I20260927 11:16:45.646431 18439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:45.647095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 18439
I20260927 11:16:45.647224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:16:45.647416 392 external_mini_cluster.cc:1428] Setting key 99a38d00543d820edd1d58a2c59324b9
I20260927 11:16:45.647433 18507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:45.650635 18507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79: Bootstrap starting.
I20260927 11:16:45.651253 18507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:45.651561 18507 log.cc:824] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:45.652256 18507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79: No bootstrap required, opened a new log
Sep 27 11:16:45 dist-test-slave-9hq9 krb5kdc[18426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:16:45.654779 18507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29bd2a32ca30475f9e89b93b2b94dc79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45393 } }
I20260927 11:16:45.654958 18507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:45.654997 18507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29bd2a32ca30475f9e89b93b2b94dc79, State: Initialized, Role: FOLLOWER
I20260927 11:16:45.655128 18507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29bd2a32ca30475f9e89b93b2b94dc79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45393 } }
I20260927 11:16:45.655215 18507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:45.655252 18507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:45.655301 18507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:45.655968 18507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29bd2a32ca30475f9e89b93b2b94dc79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45393 } }
I20260927 11:16:45.656069 18507 leader_election.cc:302] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29bd2a32ca30475f9e89b93b2b94dc79; no voters:
I20260927 11:16:45.656229 18507 leader_election.cc:288] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:16:45.656289 18512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:45.656484 18512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [term 1 LEADER]: Becoming Leader. State: Replica: 29bd2a32ca30475f9e89b93b2b94dc79, State: Running, Role: LEADER
I20260927 11:16:45.656591 18512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29bd2a32ca30475f9e89b93b2b94dc79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45393 } }
I20260927 11:16:45.656479 18507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:45.657096 18514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29bd2a32ca30475f9e89b93b2b94dc79. Latest consensus state: current_term: 1 leader_uuid: "29bd2a32ca30475f9e89b93b2b94dc79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29bd2a32ca30475f9e89b93b2b94dc79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45393 } } }
I20260927 11:16:45.657184 18514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:45.657084 18513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29bd2a32ca30475f9e89b93b2b94dc79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29bd2a32ca30475f9e89b93b2b94dc79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45393 } } }
I20260927 11:16:45.657563 18513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:45.657864 18517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:45.658409 18517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:16:45.658632 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:45.647895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.0.98.62:45393)
0927 11:16:45.648077 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:45.648085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:45.648107 (+ 22us) server_negotiation.cc:407] Connection header received
0927 11:16:45.648479 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:45.648484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:45.648525 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:45.648627 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:45.649406 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:45.650324 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:45.651364 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:45.651806 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:45.651883 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:45.654466 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:45.654470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:45.654471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:45.654483 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:45.656445 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:45.656844 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:45.656847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:45.656847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:45.656955 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:45.657193 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:45.657194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:45.657194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:45.657519 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:45.658161 (+ 642us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:45.658236 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 11:16:45.658415 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260927 11:16:45.660107 18517 catalog_manager.cc:1347] Generated new cluster ID: 27481807a8f74997bfa9c00cb782ac02
I20260927 11:16:45.660156 18517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:16:45.676898 18517 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:16:45.677572 18517 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:45.683408 18517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79: Generated new TSK 0
I20260927 11:16:45.683560 18517 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:16:45.715869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45393
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:45.819284 18533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:45.819504 18533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:45.819538 18533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:45.820675 18533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:45.820732 18533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:45.820775 18533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:16:45.822220 18533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:45.822471 18533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:45.822674 18533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:45.827258 18541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:45.827287 18539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:45.827258 18538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:45.827459 18533 server_base.cc:1034] running on GCE node
I20260927 11:16:45.827646 18533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:45.827816 18533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:45.828945 18533 hybrid_clock.cc:648] HybridClock initialized: now 1790507805828923 us; error 39 us; skew 500 ppm
Sep 27 11:16:45 dist-test-slave-9hq9 krb5kdc[18426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:45.831362 18533 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:16:45.831717 18533 webserver.cc:460] Webserver started at http://127.0.98.1:44927/ using document root <none> and password file <none>
I20260927 11:16:45.831887 18533 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:45.831952 18533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:45.832044 18533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:45.832937 18533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "e26bd12e15334b43b62954b1cbd94083"
format_stamp: "Formatted at 2026-09-27 11:16:45 on dist-test-slave-9hq9"
server_key: "648fe3436bdb72b06e9f7c27c4100cc8"
server_key_iv: "781642b9d6bf3c7c2a1adc0ca3ec18bb"
server_key_version: "encryptionkey@0"
I20260927 11:16:45.833221 18533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e26bd12e15334b43b62954b1cbd94083"
format_stamp: "Formatted at 2026-09-27 11:16:45 on dist-test-slave-9hq9"
server_key: "648fe3436bdb72b06e9f7c27c4100cc8"
server_key_iv: "781642b9d6bf3c7c2a1adc0ca3ec18bb"
server_key_version: "encryptionkey@0"
I20260927 11:16:45.834471 18533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:16:45.835086 18548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:45.835245 18533 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:45.835312 18533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "e26bd12e15334b43b62954b1cbd94083"
format_stamp: "Formatted at 2026-09-27 11:16:45 on dist-test-slave-9hq9"
server_key: "648fe3436bdb72b06e9f7c27c4100cc8"
server_key_iv: "781642b9d6bf3c7c2a1adc0ca3ec18bb"
server_key_version: "encryptionkey@0"
I20260927 11:16:45.835361 18533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:45.865161 18533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:45.867430 18533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:45.867538 18533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:45.867692 18533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:45.867983 18533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:45.868016 18533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:45.868070 18533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:45.868095 18533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:45.873595 18533 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37715
I20260927 11:16:45.873667 18661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37715 every 8 connection(s)
I20260927 11:16:45.873961 18533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:16:45 dist-test-slave-9hq9 krb5kdc[18426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:16:45.881748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 18533
I20260927 11:16:45.881873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:16:45.882100 392 external_mini_cluster.cc:1428] Setting key 4ea5c96941f1589a44b5560dee3a26e2
I20260927 11:16:45.882243 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:45.874634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32887 (local address 127.0.98.62:45393)
0927 11:16:45.874710 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:45.874712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:45.874889 (+ 177us) server_negotiation.cc:407] Connection header received
0927 11:16:45.875739 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:45.875740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:45.875749 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:45.875796 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:45.876327 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:45.876811 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:45.877717 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:45.877904 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:45.877953 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:45.879835 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:45.879836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:45.879836 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:45.879842 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:45.880929 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:45.881252 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:45.881252 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:45.881253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:45.881298 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:45.881534 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:45.881535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:45.881535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:45.881692 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:45.881767 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:45.882073 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 11:16:45.882157 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 11:16:45.882453 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:45.874599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45393 (local address 127.0.98.1:32887)
0927 11:16:45.874783 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:45.874815 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:45.875657 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:45.875855 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:45.875858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:45.876090 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:45.876260 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:45.876264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:45.876884 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:45.876884 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:45.877648 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:45.877650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:45.877977 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:45.877977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:45.878116 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:45.878511 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:45.878519 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:45.879752 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:45.880995 (+ 1243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:45.881000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:45.881002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:45.881192 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:45.881353 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:45.881354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:45.881354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:45.881472 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:45.881760 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:45.881764 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:45.881926 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 11:16:45.882025 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 11:16:45.882222 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
I20260927 11:16:45.882956 18662 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45393
I20260927 11:16:45.883066 18662 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:45.883229 18662 heartbeater.cc:511] Master 127.0.98.62:45393 requested a full tablet report, sending...
I20260927 11:16:45.883754 18471 ts_manager.cc:194] Registered new tserver with Master: e26bd12e15334b43b62954b1cbd94083 (127.0.98.1:37715)
I20260927 11:16:45.884543 18471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32887
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:16:45.904762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45393
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:45.982537 18667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:45.982725 18667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:45.982748 18667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:45.983897 18667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:45.983939 18667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:45.983994 18667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:16:45.985489 18667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:45.985714 18667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:45.985936 18667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:45.990947 18675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:45.990958 18672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:45.990942 18673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:45.991246 18667 server_base.cc:1034] running on GCE node
I20260927 11:16:45.991456 18667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:45.991683 18667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:45.992836 18667 hybrid_clock.cc:648] HybridClock initialized: now 1790507805992831 us; error 37 us; skew 500 ppm
Sep 27 11:16:45 dist-test-slave-9hq9 krb5kdc[18426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:45.995219 18667 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:16:45.995569 18667 webserver.cc:460] Webserver started at http://127.0.98.2:37003/ using document root <none> and password file <none>
I20260927 11:16:45.995754 18667 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:45.995791 18667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:45.995872 18667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:45.996781 18667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "e857a9f15e464cd0aa7b76cd96fcd295"
format_stamp: "Formatted at 2026-09-27 11:16:45 on dist-test-slave-9hq9"
server_key: "b26040765bc324aa1cfa0f6b384ee07b"
server_key_iv: "2c41751b0b786a8e9919dd39e87bfe6f"
server_key_version: "encryptionkey@0"
I20260927 11:16:45.997061 18667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e857a9f15e464cd0aa7b76cd96fcd295"
format_stamp: "Formatted at 2026-09-27 11:16:45 on dist-test-slave-9hq9"
server_key: "b26040765bc324aa1cfa0f6b384ee07b"
server_key_iv: "2c41751b0b786a8e9919dd39e87bfe6f"
server_key_version: "encryptionkey@0"
I20260927 11:16:45.998234 18667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:16:45.998849 18682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:45.999001 18667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:45.999089 18667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "e857a9f15e464cd0aa7b76cd96fcd295"
format_stamp: "Formatted at 2026-09-27 11:16:45 on dist-test-slave-9hq9"
server_key: "b26040765bc324aa1cfa0f6b384ee07b"
server_key_iv: "2c41751b0b786a8e9919dd39e87bfe6f"
server_key_version: "encryptionkey@0"
I20260927 11:16:45.999155 18667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:46.022133 18667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:46.023910 18667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:46.024073 18667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:46.024293 18667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:46.024574 18667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:46.024606 18667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:46.024627 18667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:46.024672 18667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:46.030045 18667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44635
I20260927 11:16:46.030135 18795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44635 every 8 connection(s)
I20260927 11:16:46.030422 18667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:16:46 dist-test-slave-9hq9 krb5kdc[18426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:16:46.039281 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.031101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54547 (local address 127.0.98.62:45393)
0927 11:16:46.031161 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:46.031163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:46.031350 (+ 187us) server_negotiation.cc:407] Connection header received
0927 11:16:46.032267 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:46.032268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:46.032280 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:46.032333 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:46.032892 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.033387 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.034411 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.034627 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.034700 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.036658 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:46.036660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:46.036660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:46.036668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:46.037923 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:46.038349 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:46.038350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:46.038350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:46.038411 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:46.038647 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:46.038648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:46.038648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:46.038762 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:46.038838 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:46.039101 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 11:16:46.039193 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 11:16:46.039469 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.031109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45393 (local address 127.0.98.2:54547)
0927 11:16:46.031239 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:46.031270 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:46.032160 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:46.032398 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:46.032406 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:46.032631 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:46.032794 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.032798 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.033477 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.033478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.034327 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.034329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.034717 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.034718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.034840 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.035307 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:46.035314 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:46.036562 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:46.038009 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:46.038016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:46.038018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:46.038260 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:46.038477 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:46.038478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:46.038478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:46.038577 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:46.038828 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:46.038834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:46.038977 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 11:16:46.039080 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 11:16:46.039225 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260927 11:16:46.039529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 18667
I20260927 11:16:46.039628 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:16:46.039846 392 external_mini_cluster.cc:1428] Setting key 984a6a5c71e90e8036d025411264ca51
I20260927 11:16:46.039887 18796 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45393
I20260927 11:16:46.039974 18796 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:46.040170 18796 heartbeater.cc:511] Master 127.0.98.62:45393 requested a full tablet report, sending...
I20260927 11:16:46.040565 18471 ts_manager.cc:194] Registered new tserver with Master: e857a9f15e464cd0aa7b76cd96fcd295 (127.0.98.2:44635)
I20260927 11:16:46.041047 18471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54547
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:16:46.062413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45393
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:46.139904 18801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:46.140082 18801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:46.140105 18801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:46.141119 18801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:46.141150 18801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:46.141173 18801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:16:46.142531 18801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:46.142683 18801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:46.142833 18801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:46.147459 18807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:46.147501 18806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:46.147501 18809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:46.147684 18801 server_base.cc:1034] running on GCE node
I20260927 11:16:46.147887 18801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:46.148065 18801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:46.149199 18801 hybrid_clock.cc:648] HybridClock initialized: now 1790507806149186 us; error 29 us; skew 500 ppm
Sep 27 11:16:46 dist-test-slave-9hq9 krb5kdc[18426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:46.151536 18801 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:16:46.151878 18801 webserver.cc:460] Webserver started at http://127.0.98.3:35169/ using document root <none> and password file <none>
I20260927 11:16:46.152047 18801 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:46.152083 18801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:46.152212 18801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:46.153102 18801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "13b1b54544994cc1b4dc088d34693ade"
format_stamp: "Formatted at 2026-09-27 11:16:46 on dist-test-slave-9hq9"
server_key: "157ed969326b7c2bee18824033bab27f"
server_key_iv: "06f373ff1a5b29de908b383200e5ced9"
server_key_version: "encryptionkey@0"
I20260927 11:16:46.153421 18801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "13b1b54544994cc1b4dc088d34693ade"
format_stamp: "Formatted at 2026-09-27 11:16:46 on dist-test-slave-9hq9"
server_key: "157ed969326b7c2bee18824033bab27f"
server_key_iv: "06f373ff1a5b29de908b383200e5ced9"
server_key_version: "encryptionkey@0"
I20260927 11:16:46.154670 18801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:16:46.155401 18816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:46.155586 18801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:16:46.155678 18801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "13b1b54544994cc1b4dc088d34693ade"
format_stamp: "Formatted at 2026-09-27 11:16:46 on dist-test-slave-9hq9"
server_key: "157ed969326b7c2bee18824033bab27f"
server_key_iv: "06f373ff1a5b29de908b383200e5ced9"
server_key_version: "encryptionkey@0"
I20260927 11:16:46.155750 18801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:46.187216 18801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:46.188890 18801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:46.188985 18801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:46.189134 18801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:46.189344 18801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:46.189371 18801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:46.189390 18801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:46.189402 18801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:46.194738 18801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38259
I20260927 11:16:46.194808 18929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38259 every 8 connection(s)
I20260927 11:16:46.195067 18801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:16:46.196791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 18801
I20260927 11:16:46.196899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:16:46.197057 392 external_mini_cluster.cc:1428] Setting key 3f54f34318415601c432a86a19909855
Sep 27 11:16:46 dist-test-slave-9hq9 krb5kdc[18426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:16:46.203812 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.195901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53291 (local address 127.0.98.62:45393)
0927 11:16:46.195950 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:46.195952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:46.195961 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:16:46.196830 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:46.196831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:46.196843 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:46.196904 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:46.197644 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.198311 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.199179 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.199345 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.199390 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.201356 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:46.201357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:46.201357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:46.201363 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:46.202654 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:46.202995 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:46.202996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:46.202996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:46.203060 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:46.203273 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:46.203274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:46.203274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:46.203396 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:46.203462 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:46.203674 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 11:16:46.203739 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:16:46.203929 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.195714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45393 (local address 127.0.98.3:53291)
0927 11:16:46.195852 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:46.195888 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:46.196728 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:46.196968 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:46.196972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:46.197308 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:46.197554 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.197560 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.198382 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.198383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.199114 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.199116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.199409 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.199409 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.199496 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.199878 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:46.199886 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:46.201281 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:46.202744 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:46.202749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:46.202751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:46.202937 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:46.203124 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:46.203124 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:46.203124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:46.203214 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:46.203476 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:46.203480 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:46.203573 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 11:16:46.203645 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 11:16:46.203752 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260927 11:16:46.204355 18930 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45393
I20260927 11:16:46.204445 18930 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:46.204595 18930 heartbeater.cc:511] Master 127.0.98.62:45393 requested a full tablet report, sending...
I20260927 11:16:46.204957 18471 ts_manager.cc:194] Registered new tserver with Master: 13b1b54544994cc1b4dc088d34693ade (127.0.98.3:38259)
I20260927 11:16:46.205271 18471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53291
I20260927 11:16:46.208645 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:16:46.215400 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.209779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43640 (local address 127.0.98.62:45393)
0927 11:16:46.209855 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:46.209857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:46.209924 (+ 67us) server_negotiation.cc:407] Connection header received
0927 11:16:46.209957 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:46.209958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:46.209965 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:46.210007 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:46.210585 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.211020 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.211672 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.211848 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.211895 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.212970 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:46.212971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:46.212971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:46.212976 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:46.214150 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:46.214529 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:46.214530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:46.214530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:46.214584 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:46.214881 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:46.214882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:46.214882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:46.215002 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:46.215086 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:46.215212 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 11:16:46.215300 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 11:16:46.217142 18471 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:16:46.218742 18471 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:16:46.224881 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.220734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37715 (local address 127.0.0.1:49092)
0927 11:16:46.220913 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:46.220952 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:46.221013 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:46.221460 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:46.221461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:46.221462 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:46.221499 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.221500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.222859 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.222861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.223667 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.223670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.224561 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.224562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.224639 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.224656 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 11:16:46.224698 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 11:16:46.224746 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":47,"threads_started":1}
I20260927 11:16:46.224903 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.220907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44542 (local address 127.0.98.2:44635)
0927 11:16:46.221086 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:46.221090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:46.221141 (+ 51us) server_negotiation.cc:407] Connection header received
0927 11:16:46.221228 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:46.221230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:46.221245 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:46.221322 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:46.221596 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.222712 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.223962 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.224609 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.224666 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.224728 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:46.224795 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 11:16:46.224829 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20260927 11:16:46.224887 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.220740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44635 (local address 127.0.0.1:44542)
0927 11:16:46.221075 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:46.221085 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:46.221151 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:46.221346 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:46.221349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:46.221351 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:46.221411 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.221412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.222835 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.222836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.223890 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.223892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.224679 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.224680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.224737 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.224740 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:46.224786 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 11:16:46.224818 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260927 11:16:46.225220 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.220940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49092 (local address 127.0.98.1:37715)
0927 11:16:46.221280 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:46.221285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:46.221295 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:16:46.221333 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:46.221335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:46.221349 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:46.221454 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:46.221606 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.222722 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.223756 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.224462 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.224815 (+ 353us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.224873 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:46.224895 (+ 22us) server_negotiation.cc:299] Negotiation successful
0927 11:16:46.225126 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":89,"threads_started":1}
I20260927 11:16:46.225221 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.221159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.0.98.3:38259)
0927 11:16:46.221289 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:46.221293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:46.221301 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:16:46.221332 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:46.221334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:46.221349 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:46.221454 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:46.221679 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.222489 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.223915 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.224465 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.224551 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.224604 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:46.225095 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 11:16:46.225139 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
I20260927 11:16:46.225309 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.220788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38259 (local address 127.0.0.1:51290)
0927 11:16:46.221112 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:46.221119 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:46.221154 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:46.221541 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:46.221542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:46.221543 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:46.221586 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.221588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.222587 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.222588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.223819 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.223822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.224564 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.224565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.224662 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.224667 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:16:46.224720 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 11:16:46.224764 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":69,"threads_started":1}
I20260927 11:16:46.225687 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 1d677e9a75e8427b8fc93c22f2763a2a (DEFAULT_TABLE table=test-table [id=ac512e5b267b49e289d9690463dd48c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:46.225926 18864 tablet_service.cc:1467] Processing CreateTablet for tablet 1d677e9a75e8427b8fc93c22f2763a2a (DEFAULT_TABLE table=test-table [id=ac512e5b267b49e289d9690463dd48c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:46.226040 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d677e9a75e8427b8fc93c22f2763a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:46.226224 18864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d677e9a75e8427b8fc93c22f2763a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:46.226770 18596 tablet_service.cc:1467] Processing CreateTablet for tablet 1d677e9a75e8427b8fc93c22f2763a2a (DEFAULT_TABLE table=test-table [id=ac512e5b267b49e289d9690463dd48c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:46.227052 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d677e9a75e8427b8fc93c22f2763a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:46.229254 18948 tablet_bootstrap.cc:492] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295: Bootstrap starting.
I20260927 11:16:46.229589 18949 tablet_bootstrap.cc:492] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083: Bootstrap starting.
I20260927 11:16:46.230007 18948 tablet_bootstrap.cc:654] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:46.230225 18947 tablet_bootstrap.cc:492] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade: Bootstrap starting.
I20260927 11:16:46.230273 18949 tablet_bootstrap.cc:654] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:46.230612 18948 log.cc:824] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:46.230782 18947 tablet_bootstrap.cc:654] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:46.230870 18949 log.cc:824] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:46.231351 18947 log.cc:824] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:46.231420 18948 tablet_bootstrap.cc:492] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295: No bootstrap required, opened a new log
I20260927 11:16:46.231473 18948 ts_tablet_manager.cc:1397] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:16:46.231511 18949 tablet_bootstrap.cc:492] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083: No bootstrap required, opened a new log
I20260927 11:16:46.231580 18949 ts_tablet_manager.cc:1397] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:16:46.232120 18947 tablet_bootstrap.cc:492] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade: No bootstrap required, opened a new log
I20260927 11:16:46.232179 18947 ts_tablet_manager.cc:1397] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:16:46.233178 18949 raft_consensus.cc:348] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26bd12e15334b43b62954b1cbd94083" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37715 } } peers { permanent_uuid: "e857a9f15e464cd0aa7b76cd96fcd295" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44635 } } peers { permanent_uuid: "13b1b54544994cc1b4dc088d34693ade" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38259 } }
I20260927 11:16:46.233332 18949 raft_consensus.cc:374] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:46.233376 18949 raft_consensus.cc:729] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26bd12e15334b43b62954b1cbd94083, State: Initialized, Role: FOLLOWER
I20260927 11:16:46.233361 18948 raft_consensus.cc:348] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26bd12e15334b43b62954b1cbd94083" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37715 } } peers { permanent_uuid: "e857a9f15e464cd0aa7b76cd96fcd295" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44635 } } peers { permanent_uuid: "13b1b54544994cc1b4dc088d34693ade" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38259 } }
I20260927 11:16:46.233528 18948 raft_consensus.cc:374] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:46.233479 18949 consensus_queue.cc:260] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26bd12e15334b43b62954b1cbd94083" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37715 } } peers { permanent_uuid: "e857a9f15e464cd0aa7b76cd96fcd295" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44635 } } peers { permanent_uuid: "13b1b54544994cc1b4dc088d34693ade" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38259 } }
I20260927 11:16:46.233562 18948 raft_consensus.cc:729] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e857a9f15e464cd0aa7b76cd96fcd295, State: Initialized, Role: FOLLOWER
I20260927 11:16:46.233655 18948 consensus_queue.cc:260] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26bd12e15334b43b62954b1cbd94083" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37715 } } peers { permanent_uuid: "e857a9f15e464cd0aa7b76cd96fcd295" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44635 } } peers { permanent_uuid: "13b1b54544994cc1b4dc088d34693ade" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38259 } }
I20260927 11:16:46.233798 18949 ts_tablet_manager.cc:1428] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:16:46.233917 18662 heartbeater.cc:503] Master 127.0.98.62:45393 was elected leader, sending a full tablet report...
I20260927 11:16:46.233925 18948 ts_tablet_manager.cc:1428] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:16:46.234017 18796 heartbeater.cc:503] Master 127.0.98.62:45393 was elected leader, sending a full tablet report...
I20260927 11:16:46.234031 18947 raft_consensus.cc:348] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26bd12e15334b43b62954b1cbd94083" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37715 } } peers { permanent_uuid: "e857a9f15e464cd0aa7b76cd96fcd295" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44635 } } peers { permanent_uuid: "13b1b54544994cc1b4dc088d34693ade" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38259 } }
I20260927 11:16:46.234200 18947 raft_consensus.cc:374] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:46.234249 18947 raft_consensus.cc:729] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b1b54544994cc1b4dc088d34693ade, State: Initialized, Role: FOLLOWER
I20260927 11:16:46.234345 18947 consensus_queue.cc:260] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26bd12e15334b43b62954b1cbd94083" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37715 } } peers { permanent_uuid: "e857a9f15e464cd0aa7b76cd96fcd295" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44635 } } peers { permanent_uuid: "13b1b54544994cc1b4dc088d34693ade" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38259 } }
I20260927 11:16:46.234584 18947 ts_tablet_manager.cc:1428] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:16:46.234602 18930 heartbeater.cc:503] Master 127.0.98.62:45393 was elected leader, sending a full tablet report...
W20260927 11:16:46.280979 18797 tablet.cc:2369] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:16:46.374915 18663 tablet.cc:2369] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 11:16:46.445709 18931 tablet.cc:2369] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:16:46.552366 18954 raft_consensus.cc:482] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:16:46.552495 18954 raft_consensus.cc:504] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26bd12e15334b43b62954b1cbd94083" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37715 } } peers { permanent_uuid: "e857a9f15e464cd0aa7b76cd96fcd295" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44635 } } peers { permanent_uuid: "13b1b54544994cc1b4dc088d34693ade" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38259 } }
I20260927 11:16:46.552891 18954 leader_election.cc:288] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e26bd12e15334b43b62954b1cbd94083 (127.0.98.1:37715), 13b1b54544994cc1b4dc088d34693ade (127.0.98.3:38259)
I20260927 11:16:46.555780 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.553122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50747 (local address 127.0.98.1:37715)
0927 11:16:46.553169 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:46.553171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:46.553276 (+ 105us) server_negotiation.cc:407] Connection header received
0927 11:16:46.553375 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:46.553376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:46.553393 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:46.553458 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:46.553618 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.554163 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.555073 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.555444 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.555497 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.555505 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:46.555663 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 11:16:46.555711 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:16:46.555786 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.553064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37715 (local address 127.0.98.2:50747)
0927 11:16:46.553192 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:46.553202 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:46.553274 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:46.553467 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:46.553468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:46.553469 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:46.553548 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.553550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.554234 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.554235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.554985 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.554987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.555520 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.555522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.555590 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.555599 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 11:16:46.555640 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 11:16:46.555669 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":2,"thread_start_us":65,"threads_started":1}
I20260927 11:16:46.555830 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.553134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49133 (local address 127.0.98.3:38259)
0927 11:16:46.553197 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:46.553199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:46.553232 (+ 33us) server_negotiation.cc:407] Connection header received
0927 11:16:46.553374 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:46.553375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:46.553384 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:46.553458 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:46.553658 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.554166 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.555106 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.555457 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.555544 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.555553 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:46.555720 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 11:16:46.555760 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 11:16:46.556054 18957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.553055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38259 (local address 127.0.98.2:49133)
0927 11:16:46.553174 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:46.553183 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:46.553264 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:46.553464 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:46.553465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:46.553467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:46.553566 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.553567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.554254 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.554255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.554995 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.554997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.555558 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.555559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.555642 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.555647 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 11:16:46.555704 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 11:16:46.555737 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":18,"thread_start_us":57,"threads_started":1}
I20260927 11:16:46.556239 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d677e9a75e8427b8fc93c22f2763a2a" candidate_uuid: "e857a9f15e464cd0aa7b76cd96fcd295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26bd12e15334b43b62954b1cbd94083" is_pre_election: true
I20260927 11:16:46.556389 18616 raft_consensus.cc:2454] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e857a9f15e464cd0aa7b76cd96fcd295 in term 0.
I20260927 11:16:46.556490 18884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d677e9a75e8427b8fc93c22f2763a2a" candidate_uuid: "e857a9f15e464cd0aa7b76cd96fcd295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b1b54544994cc1b4dc088d34693ade" is_pre_election: true
I20260927 11:16:46.556643 18884 raft_consensus.cc:2454] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e857a9f15e464cd0aa7b76cd96fcd295 in term 0.
I20260927 11:16:46.556651 18686 leader_election.cc:302] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e26bd12e15334b43b62954b1cbd94083, e857a9f15e464cd0aa7b76cd96fcd295; no voters:
I20260927 11:16:46.556783 18954 raft_consensus.cc:2781] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:16:46.556854 18954 raft_consensus.cc:482] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:16:46.556879 18954 raft_consensus.cc:3037] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:46.557706 18954 raft_consensus.cc:504] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26bd12e15334b43b62954b1cbd94083" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37715 } } peers { permanent_uuid: "e857a9f15e464cd0aa7b76cd96fcd295" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44635 } } peers { permanent_uuid: "13b1b54544994cc1b4dc088d34693ade" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38259 } }
I20260927 11:16:46.557843 18954 leader_election.cc:288] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [CANDIDATE]: Term 1 election: Requested vote from peers e26bd12e15334b43b62954b1cbd94083 (127.0.98.1:37715), 13b1b54544994cc1b4dc088d34693ade (127.0.98.3:38259)
I20260927 11:16:46.558007 18884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d677e9a75e8427b8fc93c22f2763a2a" candidate_uuid: "e857a9f15e464cd0aa7b76cd96fcd295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b1b54544994cc1b4dc088d34693ade"
I20260927 11:16:46.558017 18616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d677e9a75e8427b8fc93c22f2763a2a" candidate_uuid: "e857a9f15e464cd0aa7b76cd96fcd295" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26bd12e15334b43b62954b1cbd94083"
I20260927 11:16:46.558099 18884 raft_consensus.cc:3037] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:46.558113 18616 raft_consensus.cc:3037] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:46.558933 18616 raft_consensus.cc:2454] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e857a9f15e464cd0aa7b76cd96fcd295 in term 1.
I20260927 11:16:46.558933 18884 raft_consensus.cc:2454] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e857a9f15e464cd0aa7b76cd96fcd295 in term 1.
I20260927 11:16:46.559167 18684 leader_election.cc:302] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b1b54544994cc1b4dc088d34693ade, e857a9f15e464cd0aa7b76cd96fcd295; no voters:
I20260927 11:16:46.559267 18954 raft_consensus.cc:2781] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:46.559427 18954 raft_consensus.cc:686] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [term 1 LEADER]: Becoming Leader. State: Replica: e857a9f15e464cd0aa7b76cd96fcd295, State: Running, Role: LEADER
I20260927 11:16:46.559520 18954 consensus_queue.cc:237] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26bd12e15334b43b62954b1cbd94083" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37715 } } peers { permanent_uuid: "e857a9f15e464cd0aa7b76cd96fcd295" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44635 } } peers { permanent_uuid: "13b1b54544994cc1b4dc088d34693ade" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38259 } }
I20260927 11:16:46.560313 18470 catalog_manager.cc:5495] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 reported cstate change: term changed from 0 to 1, leader changed from <none> to e857a9f15e464cd0aa7b76cd96fcd295 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e857a9f15e464cd0aa7b76cd96fcd295" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26bd12e15334b43b62954b1cbd94083" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e857a9f15e464cd0aa7b76cd96fcd295" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13b1b54544994cc1b4dc088d34693ade" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38259 } health_report { overall_health: UNKNOWN } } }
I20260927 11:16:46.578962 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.576878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.0.98.2:44635)
0927 11:16:46.576934 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:46.576935 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:46.576943 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:16:46.576996 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:46.576996 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:46.577010 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:46.577066 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:16:46.577172 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.577606 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.578332 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.578499 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.578547 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.578661 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:16:46.578736 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:16:46.578806 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:46.578854 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 11:16:46.578883 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 11:16:46.580528 18884 raft_consensus.cc:1261] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade [term 1 FOLLOWER]: Refusing update from remote peer e857a9f15e464cd0aa7b76cd96fcd295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:16:46.580603 18616 raft_consensus.cc:1261] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083 [term 1 FOLLOWER]: Refusing update from remote peer e857a9f15e464cd0aa7b76cd96fcd295: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:16:46.580808 18954 consensus_queue.cc:1035] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13b1b54544994cc1b4dc088d34693ade" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:16:46.580936 18958 consensus_queue.cc:1035] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e26bd12e15334b43b62954b1cbd94083" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:16:46.583423 18962 mvcc.cc:205] Tried to move back new op lower bound from 7333919975751327744 to 7333919975668215808. Current Snapshot: MvccSnapshot[applied={T|T < 7333919975751327744}]
I20260927 11:16:46.585660 18471 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:16:46.585747 18471 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:16:46.586480 18471 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29bd2a32ca30475f9e89b93b2b94dc79: Sending DeleteTablet for 3 replicas of tablet 1d677e9a75e8427b8fc93c22f2763a2a
I20260927 11:16:46.586786 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d677e9a75e8427b8fc93c22f2763a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44542
I20260927 11:16:46.586769 18596 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d677e9a75e8427b8fc93c22f2763a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49092
I20260927 11:16:46.586823 18864 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d677e9a75e8427b8fc93c22f2763a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51290
I20260927 11:16:46.586982 18972 tablet_replica.cc:330] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083: stopping tablet replica
I20260927 11:16:46.587072 18972 raft_consensus.cc:2229] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:16:46.587168 18971 tablet_replica.cc:330] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295: stopping tablet replica
I20260927 11:16:46.587188 18972 raft_consensus.cc:2258] T 1d677e9a75e8427b8fc93c22f2763a2a P e26bd12e15334b43b62954b1cbd94083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:46.587329 18973 tablet_replica.cc:330] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade: stopping tablet replica
I20260927 11:16:46.587330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 18533
I20260927 11:16:46.587409 18973 raft_consensus.cc:2229] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:16:46.587491 18973 raft_consensus.cc:2258] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:46.587241 18971 raft_consensus.cc:2229] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:16:46.587958 18971 raft_consensus.cc:2258] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:46.588438 18971 ts_tablet_manager.cc:1905] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:16:46.589996 18971 ts_tablet_manager.cc:1918] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:16:46.590042 18971 log.cc:1196] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/1d677e9a75e8427b8fc93c22f2763a2a
I20260927 11:16:46.590349 18971 ts_tablet_manager.cc:1939] T 1d677e9a75e8427b8fc93c22f2763a2a P e857a9f15e464cd0aa7b76cd96fcd295: Deleting consensus metadata
I20260927 11:16:46.590735 18455 catalog_manager.cc:4841] TS e857a9f15e464cd0aa7b76cd96fcd295 (127.0.98.2:44635): tablet 1d677e9a75e8427b8fc93c22f2763a2a (table test-table [id=ac512e5b267b49e289d9690463dd48c5]) successfully deleted
W20260927 11:16:46.593751 18458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37715 (error 108)
I20260927 11:16:46.594123 18973 ts_tablet_manager.cc:1905] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:16:46.594267 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.594134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37715 (local address 127.0.0.1:49108)
0927 11:16:46.594194 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:46.594235 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 11:16:46.594380 18458 catalog_manager.cc:4584] TS e26bd12e15334b43b62954b1cbd94083 (127.0.98.1:37715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d677e9a75e8427b8fc93c22f2763a2a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37715: connect: Connection refused (error 111)
I20260927 11:16:46.594630 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 18667
I20260927 11:16:46.596490 18973 ts_tablet_manager.cc:1918] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:16:46.596563 18973 log.cc:1196] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/1d677e9a75e8427b8fc93c22f2763a2a
I20260927 11:16:46.596799 18973 ts_tablet_manager.cc:1939] T 1d677e9a75e8427b8fc93c22f2763a2a P 13b1b54544994cc1b4dc088d34693ade: Deleting consensus metadata
I20260927 11:16:46.597175 18456 catalog_manager.cc:4841] TS 13b1b54544994cc1b4dc088d34693ade (127.0.98.3:38259): tablet 1d677e9a75e8427b8fc93c22f2763a2a (table test-table [id=ac512e5b267b49e289d9690463dd48c5]) successfully deleted
I20260927 11:16:46.599380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 18801
I20260927 11:16:46.603919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 18439
2026-09-27T11:16:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2205 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T11:16:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:16:46Z Disabled control of system clock
I20260927 11:16:46.630378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41419
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:16:46.708164 18978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:46.708346 18978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:46.708365 18978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:46.709524 18978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:46.709563 18978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:46.709575 18978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:46.709589 18978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:46.709625 18978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:46.710983 18978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41419
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:46.711215 18978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:46.711509 18978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:46.716048 18986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:46.716058 18983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:46.716051 18984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:46.716310 18978 server_base.cc:1034] running on GCE node
I20260927 11:16:46.716570 18978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:46.716852 18978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:46.718006 18978 hybrid_clock.cc:648] HybridClock initialized: now 1790507806717996 us; error 27 us; skew 500 ppm
I20260927 11:16:46.719213 18978 webserver.cc:460] Webserver started at http://127.0.98.62:42355/ using document root <none> and password file <none>
I20260927 11:16:46.719439 18978 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:46.719514 18978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:46.719666 18978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:46.720625 18978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "de8f830ca9bb4fc69eb5b61be343af2f"
format_stamp: "Formatted at 2026-09-27 11:16:46 on dist-test-slave-9hq9"
server_key: "9a66d8eefadfbfe22df50cd6e1263bd7"
server_key_iv: "8687123be95f40ff31deaaee316acb1f"
server_key_version: "encryptionkey@0"
I20260927 11:16:46.720964 18978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "de8f830ca9bb4fc69eb5b61be343af2f"
format_stamp: "Formatted at 2026-09-27 11:16:46 on dist-test-slave-9hq9"
server_key: "9a66d8eefadfbfe22df50cd6e1263bd7"
server_key_iv: "8687123be95f40ff31deaaee316acb1f"
server_key_version: "encryptionkey@0"
I20260927 11:16:46.722240 18978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:16:46.722934 18992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:46.723098 18978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:46.723191 18978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "de8f830ca9bb4fc69eb5b61be343af2f"
format_stamp: "Formatted at 2026-09-27 11:16:46 on dist-test-slave-9hq9"
server_key: "9a66d8eefadfbfe22df50cd6e1263bd7"
server_key_iv: "8687123be95f40ff31deaaee316acb1f"
server_key_version: "encryptionkey@0"
I20260927 11:16:46.723273 18978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:46.740589 18978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:46.740877 18978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:46.741032 18978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:46.744915 18978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41419
I20260927 11:16:46.744944 19044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41419 every 8 connection(s)
I20260927 11:16:46.745267 18978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:46.746176 19045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:46.748351 19045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f: Bootstrap starting.
I20260927 11:16:46.748919 19045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:46.749202 19045 log.cc:824] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:46.749830 19045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f: No bootstrap required, opened a new log
I20260927 11:16:46.751813 19045 raft_consensus.cc:348] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8f830ca9bb4fc69eb5b61be343af2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41419 } }
I20260927 11:16:46.752043 19045 raft_consensus.cc:374] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:46.752164 19045 raft_consensus.cc:729] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de8f830ca9bb4fc69eb5b61be343af2f, State: Initialized, Role: FOLLOWER
I20260927 11:16:46.752381 19045 consensus_queue.cc:260] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8f830ca9bb4fc69eb5b61be343af2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41419 } }
I20260927 11:16:46.752453 19045 raft_consensus.cc:388] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:46.752494 19045 raft_consensus.cc:482] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:46.752547 19045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:46.753063 19045 raft_consensus.cc:504] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8f830ca9bb4fc69eb5b61be343af2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41419 } }
I20260927 11:16:46.753172 19045 leader_election.cc:302] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de8f830ca9bb4fc69eb5b61be343af2f; no voters:
I20260927 11:16:46.753372 19045 leader_election.cc:288] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:16:46.753443 19048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:46.753608 19048 raft_consensus.cc:686] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [term 1 LEADER]: Becoming Leader. State: Replica: de8f830ca9bb4fc69eb5b61be343af2f, State: Running, Role: LEADER
I20260927 11:16:46.753628 19045 sys_catalog.cc:564] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:46.753705 19048 consensus_queue.cc:237] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8f830ca9bb4fc69eb5b61be343af2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41419 } }
I20260927 11:16:46.754092 19050 sys_catalog.cc:455] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader de8f830ca9bb4fc69eb5b61be343af2f. Latest consensus state: current_term: 1 leader_uuid: "de8f830ca9bb4fc69eb5b61be343af2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8f830ca9bb4fc69eb5b61be343af2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41419 } } }
I20260927 11:16:46.754189 19050 sys_catalog.cc:458] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:46.754175 19049 sys_catalog.cc:455] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de8f830ca9bb4fc69eb5b61be343af2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de8f830ca9bb4fc69eb5b61be343af2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41419 } } }
I20260927 11:16:46.754309 19049 sys_catalog.cc:458] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:46.754387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 18978
I20260927 11:16:46.754410 19055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:46.754496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:16:46.754712 392 external_mini_cluster.cc:1428] Setting key b04cf2c4d0f595c807df26fccb0c11fd
I20260927 11:16:46.754879 19055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:16:46.756681 19055 catalog_manager.cc:1347] Generated new cluster ID: cd0b687a5d534df3a294e03124612b85
I20260927 11:16:46.756732 19055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:16:46.761139 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.755119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.0.98.62:41419)
0927 11:16:46.755739 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:46.755748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:46.755776 (+ 28us) server_negotiation.cc:407] Connection header received
0927 11:16:46.755880 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:46.755888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:46.755983 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:46.756122 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:46.756488 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.757834 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.758977 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.759557 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.759660 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.760364 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:46.760368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:46.760368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:46.760563 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:46.760599 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:46.760612 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:46.760670 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:46.760769 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 11:16:46.760837 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":76,"threads_started":1}
I20260927 11:16:46.771490 19055 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:16:46.771965 19055 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:46.780947 19055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f: Generated new TSK 0
I20260927 11:16:46.781118 19055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:16:46.789805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41419
--builtin_ntp_servers=127.0.98.20:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:16:46.868461 19069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:46.868665 19069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:46.868700 19069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:46.869807 19069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:46.869863 19069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:46.869903 19069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:16:46.871263 19069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:46.871500 19069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:46.871764 19069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:46.877048 19075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:46.877140 19069 server_base.cc:1034] running on GCE node
W20260927 11:16:46.877058 19074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:46.877058 19077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:46.877485 19069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:46.877727 19069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:46.878890 19069 hybrid_clock.cc:648] HybridClock initialized: now 1790507806878875 us; error 25 us; skew 500 ppm
I20260927 11:16:46.885480 19069 webserver.cc:460] Webserver started at http://127.0.98.1:37909/ using document root <none> and password file <none>
I20260927 11:16:46.885704 19069 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:46.885772 19069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:46.885926 19069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:46.886998 19069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "cfec9fef2a5747ef93d7f62f1cf07621"
format_stamp: "Formatted at 2026-09-27 11:16:46 on dist-test-slave-9hq9"
server_key: "d0b00bb285d5d87c2b63b04da38790dc"
server_key_iv: "e0d7c28eaffd88ec45a2e6288a47b934"
server_key_version: "encryptionkey@0"
I20260927 11:16:46.887349 19069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cfec9fef2a5747ef93d7f62f1cf07621"
format_stamp: "Formatted at 2026-09-27 11:16:46 on dist-test-slave-9hq9"
server_key: "d0b00bb285d5d87c2b63b04da38790dc"
server_key_iv: "e0d7c28eaffd88ec45a2e6288a47b934"
server_key_version: "encryptionkey@0"
I20260927 11:16:46.888602 19069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:16:46.889266 19083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:46.889432 19069 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:46.889525 19069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "cfec9fef2a5747ef93d7f62f1cf07621"
format_stamp: "Formatted at 2026-09-27 11:16:46 on dist-test-slave-9hq9"
server_key: "d0b00bb285d5d87c2b63b04da38790dc"
server_key_iv: "e0d7c28eaffd88ec45a2e6288a47b934"
server_key_version: "encryptionkey@0"
I20260927 11:16:46.889609 19069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:46.902904 19069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:46.903154 19069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:46.903291 19069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:46.903504 19069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:46.903806 19069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:46.903860 19069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:46.903891 19069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:46.903918 19069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:46.909308 19069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44853
I20260927 11:16:46.909341 19196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44853 every 8 connection(s)
I20260927 11:16:46.909641 19069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:16:46.914191 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.910330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44313 (local address 127.0.98.62:41419)
0927 11:16:46.910373 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:46.910375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:46.910583 (+ 208us) server_negotiation.cc:407] Connection header received
0927 11:16:46.911034 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:46.911035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:46.911046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:46.911093 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:46.911680 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.912152 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.913033 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.913219 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.913273 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.913921 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:46.913923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:46.913923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:46.913927 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:46.913940 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:46.913944 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:46.913986 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:46.914104 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 11:16:46.914136 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:16:46.914404 19199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:46.910313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41419 (local address 127.0.98.1:44313)
0927 11:16:46.910466 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:46.910495 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:46.910954 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:46.911146 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:46.911150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:46.911157 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:46.911616 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.911620 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.912227 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.912228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.912965 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:46.912966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:46.913292 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:46.913293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:46.913392 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:46.913799 (+ 407us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:16:46.913800 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:46.913824 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:16:46.913845 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:46.913998 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:46.914002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:46.914004 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:16:46.914104 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 11:16:46.914184 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260927 11:16:46.914683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 19069
I20260927 11:16:46.914780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:16:46.914923 19197 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41419
I20260927 11:16:46.914994 392 external_mini_cluster.cc:1428] Setting key fa9a2198affff25601499a6789adbaf6
I20260927 11:16:46.915022 19197 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:46.915179 19197 heartbeater.cc:511] Master 127.0.98.62:41419 requested a full tablet report, sending...
I20260927 11:16:46.915688 19008 ts_manager.cc:194] Registered new tserver with Master: cfec9fef2a5747ef93d7f62f1cf07621 (127.0.98.1:44853)
I20260927 11:16:46.915937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41419
--builtin_ntp_servers=127.0.98.20:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 11:16:46.916397 19008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:44313
W20260927 11:16:46.993440 19200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:46.993616 19200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:46.993637 19200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:46.994693 19200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:46.994726 19200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:46.994750 19200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:16:46.995962 19200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:46.996136 19200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:46.996361 19200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:47.000998 19206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:47.001047 19208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:47.001015 19205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:47.001194 19200 server_base.cc:1034] running on GCE node
I20260927 11:16:47.001395 19200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:47.001632 19200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:47.002750 19200 hybrid_clock.cc:648] HybridClock initialized: now 1790507807002730 us; error 40 us; skew 500 ppm
I20260927 11:16:47.003806 19200 webserver.cc:460] Webserver started at http://127.0.98.2:40109/ using document root <none> and password file <none>
I20260927 11:16:47.003990 19200 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:47.004029 19200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:47.004119 19200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:47.005000 19200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "a8cf562ab87f41f288043da78629f273"
format_stamp: "Formatted at 2026-09-27 11:16:47 on dist-test-slave-9hq9"
server_key: "972e0ec43a34845dce9421e190d70e6b"
server_key_iv: "42c7ad3cf1bfac5cf0e7c0593be07b38"
server_key_version: "encryptionkey@0"
I20260927 11:16:47.005276 19200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a8cf562ab87f41f288043da78629f273"
format_stamp: "Formatted at 2026-09-27 11:16:47 on dist-test-slave-9hq9"
server_key: "972e0ec43a34845dce9421e190d70e6b"
server_key_iv: "42c7ad3cf1bfac5cf0e7c0593be07b38"
server_key_version: "encryptionkey@0"
I20260927 11:16:47.006453 19200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:16:47.007076 19214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:47.007242 19200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:16:47.007319 19200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "a8cf562ab87f41f288043da78629f273"
format_stamp: "Formatted at 2026-09-27 11:16:47 on dist-test-slave-9hq9"
server_key: "972e0ec43a34845dce9421e190d70e6b"
server_key_iv: "42c7ad3cf1bfac5cf0e7c0593be07b38"
server_key_version: "encryptionkey@0"
I20260927 11:16:47.007367 19200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:47.032894 19200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:47.033136 19200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:47.033234 19200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:47.033401 19200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:47.033716 19200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:47.033749 19200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:47.033802 19200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:47.033828 19200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:47.039225 19200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45863
I20260927 11:16:47.039312 19327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45863 every 8 connection(s)
I20260927 11:16:47.039604 19200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:16:47.040444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 19200
I20260927 11:16:47.040558 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:16:47.040756 392 external_mini_cluster.cc:1428] Setting key bd0424ee101eae77e4be0bcbbafd2441
I20260927 11:16:47.041761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41419
--builtin_ntp_servers=127.0.98.20:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 11:16:47.045629 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.040274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52219 (local address 127.0.98.62:41419)
0927 11:16:47.040354 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:47.040357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:47.040573 (+ 216us) server_negotiation.cc:407] Connection header received
0927 11:16:47.041173 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:47.041174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:47.041186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:47.041237 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:47.041924 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.042717 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.044176 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.044373 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.044451 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.045349 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:47.045350 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:47.045350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:47.045356 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:47.045374 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:47.045377 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:47.045436 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:47.045517 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 11:16:47.045553 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 11:16:47.045877 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.040264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41419 (local address 127.0.98.2:52219)
0927 11:16:47.040457 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:47.040497 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:47.041058 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:47.041303 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:47.041307 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:47.041317 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:47.041817 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:47.041823 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.042809 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.042810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:47.044068 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:47.044071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.044460 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.044461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:47.044604 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.045195 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:16:47.045195 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:47.045234 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:16:47.045257 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:47.045437 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:47.045443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:47.045446 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:47.045519 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 11:16:47.045619 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260927 11:16:47.046379 19328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41419
I20260927 11:16:47.046456 19328 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:47.046643 19328 heartbeater.cc:511] Master 127.0.98.62:41419 requested a full tablet report, sending...
I20260927 11:16:47.047024 19008 ts_manager.cc:194] Registered new tserver with Master: a8cf562ab87f41f288043da78629f273 (127.0.98.2:45863)
I20260927 11:16:47.047549 19008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52219
W20260927 11:16:47.121531 19331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:47.121824 19331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:47.121873 19331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:47.123324 19331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:47.123365 19331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:47.123389 19331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:16:47.124711 19331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:47.124953 19331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:47.125268 19331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:47.130277 19337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:47.130312 19336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:47.130312 19339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:47.130332 19331 server_base.cc:1034] running on GCE node
I20260927 11:16:47.130649 19331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:47.130834 19331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:47.131997 19331 hybrid_clock.cc:648] HybridClock initialized: now 1790507807131992 us; error 32 us; skew 500 ppm
I20260927 11:16:47.133148 19331 webserver.cc:460] Webserver started at http://127.0.98.3:34611/ using document root <none> and password file <none>
I20260927 11:16:47.133384 19331 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:47.133450 19331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:47.133594 19331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:47.134670 19331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "0569998203ec46bb924d6ddab403a0ab"
format_stamp: "Formatted at 2026-09-27 11:16:47 on dist-test-slave-9hq9"
server_key: "90e7e0f6e196cfbf93f5ea9fef0a7de3"
server_key_iv: "4f53791204ac5d41d93148e968fb9b2d"
server_key_version: "encryptionkey@0"
I20260927 11:16:47.135012 19331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0569998203ec46bb924d6ddab403a0ab"
format_stamp: "Formatted at 2026-09-27 11:16:47 on dist-test-slave-9hq9"
server_key: "90e7e0f6e196cfbf93f5ea9fef0a7de3"
server_key_iv: "4f53791204ac5d41d93148e968fb9b2d"
server_key_version: "encryptionkey@0"
I20260927 11:16:47.136271 19331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:16:47.136929 19345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:47.137099 19331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:47.137192 19331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "0569998203ec46bb924d6ddab403a0ab"
format_stamp: "Formatted at 2026-09-27 11:16:47 on dist-test-slave-9hq9"
server_key: "90e7e0f6e196cfbf93f5ea9fef0a7de3"
server_key_iv: "4f53791204ac5d41d93148e968fb9b2d"
server_key_version: "encryptionkey@0"
I20260927 11:16:47.137269 19331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:47.166939 19331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:47.167246 19331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:47.167394 19331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:47.167616 19331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:47.167945 19331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:47.167999 19331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:47.168038 19331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:47.168066 19331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:47.173478 19331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34503
I20260927 11:16:47.173569 19458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34503 every 8 connection(s)
I20260927 11:16:47.173862 19331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:16:47.176846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 19331
I20260927 11:16:47.176919 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:16:47.177111 392 external_mini_cluster.cc:1428] Setting key bacdcadccbbce595b9dfc0b5c52057c9
I20260927 11:16:47.178978 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.174593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38613 (local address 127.0.98.62:41419)
0927 11:16:47.174650 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:47.174652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:47.174907 (+ 255us) server_negotiation.cc:407] Connection header received
0927 11:16:47.175360 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:47.175362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:47.175386 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:47.175436 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:47.176023 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.176483 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.177689 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.177965 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.178022 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.178785 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:47.178786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:47.178786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:47.178790 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:47.178802 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:47.178805 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:47.178846 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:47.178905 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 11:16:47.178926 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 11:16:47.179214 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.174584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41419 (local address 127.0.98.3:38613)
0927 11:16:47.174784 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:47.174817 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:47.175278 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:47.175487 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:47.175490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:47.175497 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:47.175955 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:47.175959 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.176552 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.176553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:47.177624 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:47.177627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.178080 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.178082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:47.178215 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.178667 (+ 452us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:16:47.178668 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:47.178693 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:16:47.178714 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:47.178855 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:47.178858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:47.178860 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:16:47.178908 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 11:16:47.178981 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
I20260927 11:16:47.179687 19459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41419
I20260927 11:16:47.179790 19459 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:47.179955 19459 heartbeater.cc:511] Master 127.0.98.62:41419 requested a full tablet report, sending...
I20260927 11:16:47.180291 19008 ts_manager.cc:194] Registered new tserver with Master: 0569998203ec46bb924d6ddab403a0ab (127.0.98.3:34503)
I20260927 11:16:47.180599 19008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38613
I20260927 11:16:47.188659 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:16:47.192912 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.189948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.0.98.62:41419)
0927 11:16:47.190014 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:47.190016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:47.190149 (+ 133us) server_negotiation.cc:407] Connection header received
0927 11:16:47.190222 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:47.190223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:47.190234 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:47.190288 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:47.190454 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.190912 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.191653 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.191851 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.191931 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.192639 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:47.192641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:47.192641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:47.192645 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:47.192656 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:47.192660 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:47.192766 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:47.192828 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 11:16:47.192837 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 11:16:47.194545 19008 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:16:47.196026 19008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:16:47.202216 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.197896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34503 (local address 127.0.0.1:51216)
0927 11:16:47.197995 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:47.198030 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:47.198121 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:47.198790 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:47.198791 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:47.198793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:47.198831 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:47.198832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.199971 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.199971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:47.201092 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:47.201094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.201921 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.201922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:47.201996 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.202008 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 11:16:47.202100 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 11:16:47.202149 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":4,"thread_start_us":92,"threads_started":1}
I20260927 11:16:47.202345 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.198073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51216 (local address 127.0.98.3:34503)
0927 11:16:47.198585 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:47.198589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:47.198610 (+ 21us) server_negotiation.cc:407] Connection header received
0927 11:16:47.198660 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:47.198674 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:47.198688 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:47.198898 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:47.198940 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.199901 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.201177 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.201845 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.201902 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.201957 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:47.202160 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 11:16:47.202193 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":69,"threads_started":1}
I20260927 11:16:47.203001 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.197942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45863 (local address 127.0.0.1:49386)
0927 11:16:47.198295 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:47.198305 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:47.198356 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:47.198527 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:47.198530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:47.198532 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:47.198602 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:47.198605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.199857 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.199858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:47.201091 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:47.201094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.202782 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.202783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:47.202838 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.202841 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:47.202885 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 11:16:47.202928 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":13,"thread_start_us":55,"threads_started":1}
I20260927 11:16:47.203156 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.197902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44853 (local address 127.0.0.1:49508)
0927 11:16:47.198232 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:47.198242 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:47.198295 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:47.198527 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:47.198530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:47.198535 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:47.198602 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:47.198604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.200691 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.200692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:47.201696 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:47.201699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.202930 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.202932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:47.202991 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.202994 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:47.203044 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 11:16:47.203083 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":6,"thread_start_us":50,"threads_started":1}
I20260927 11:16:47.203136 19393 tablet_service.cc:1467] Processing CreateTablet for tablet 23f258fef43344a680f8f7d598e0923a (DEFAULT_TABLE table=test-table [id=ec109765fd82426dbf0ab65d1ee50c05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:47.203409 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.198148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49508 (local address 127.0.98.1:44853)
0927 11:16:47.198329 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:47.198333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:47.198342 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:16:47.198381 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:47.198384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:47.198399 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:47.198488 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:47.199889 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.200599 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.201785 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.202754 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.202819 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.202896 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:47.203288 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 11:16:47.203329 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20260927 11:16:47.203444 19393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f258fef43344a680f8f7d598e0923a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:47.203469 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.198130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49386 (local address 127.0.98.2:45863)
0927 11:16:47.198294 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:47.198299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:47.198348 (+ 49us) server_negotiation.cc:407] Connection header received
0927 11:16:47.198416 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:47.198418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:47.198434 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:47.198505 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:47.198694 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.199782 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.201987 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.202681 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.202778 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.202858 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:47.203261 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 11:16:47.203384 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260927 11:16:47.203933 19131 tablet_service.cc:1467] Processing CreateTablet for tablet 23f258fef43344a680f8f7d598e0923a (DEFAULT_TABLE table=test-table [id=ec109765fd82426dbf0ab65d1ee50c05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:47.204236 19131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f258fef43344a680f8f7d598e0923a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:47.205785 19473 tablet_bootstrap.cc:492] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab: Bootstrap starting.
I20260927 11:16:47.206422 19473 tablet_bootstrap.cc:654] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:47.206418 19262 tablet_service.cc:1467] Processing CreateTablet for tablet 23f258fef43344a680f8f7d598e0923a (DEFAULT_TABLE table=test-table [id=ec109765fd82426dbf0ab65d1ee50c05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:47.206738 19262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23f258fef43344a680f8f7d598e0923a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:47.206732 19473 log.cc:824] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:47.206909 19474 tablet_bootstrap.cc:492] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621: Bootstrap starting.
I20260927 11:16:47.207413 19473 tablet_bootstrap.cc:492] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab: No bootstrap required, opened a new log
I20260927 11:16:47.207494 19473 ts_tablet_manager.cc:1397] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:16:47.207576 19474 tablet_bootstrap.cc:654] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:47.207888 19474 log.cc:824] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:47.208690 19474 tablet_bootstrap.cc:492] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621: No bootstrap required, opened a new log
I20260927 11:16:47.208767 19474 ts_tablet_manager.cc:1397] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:16:47.209529 19473 raft_consensus.cc:348] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0569998203ec46bb924d6ddab403a0ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34503 } } peers { permanent_uuid: "cfec9fef2a5747ef93d7f62f1cf07621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44853 } } peers { permanent_uuid: "a8cf562ab87f41f288043da78629f273" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45863 } }
I20260927 11:16:47.209668 19473 raft_consensus.cc:374] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:47.209718 19473 raft_consensus.cc:729] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0569998203ec46bb924d6ddab403a0ab, State: Initialized, Role: FOLLOWER
I20260927 11:16:47.209861 19473 consensus_queue.cc:260] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0569998203ec46bb924d6ddab403a0ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34503 } } peers { permanent_uuid: "cfec9fef2a5747ef93d7f62f1cf07621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44853 } } peers { permanent_uuid: "a8cf562ab87f41f288043da78629f273" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45863 } }
I20260927 11:16:47.210171 19459 heartbeater.cc:503] Master 127.0.98.62:41419 was elected leader, sending a full tablet report...
I20260927 11:16:47.210282 19477 tablet_bootstrap.cc:492] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273: Bootstrap starting.
I20260927 11:16:47.210690 19473 ts_tablet_manager.cc:1428] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:16:47.210875 19477 tablet_bootstrap.cc:654] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:47.211193 19477 log.cc:824] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:47.211413 19474 raft_consensus.cc:348] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0569998203ec46bb924d6ddab403a0ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34503 } } peers { permanent_uuid: "cfec9fef2a5747ef93d7f62f1cf07621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44853 } } peers { permanent_uuid: "a8cf562ab87f41f288043da78629f273" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45863 } }
I20260927 11:16:47.211550 19474 raft_consensus.cc:374] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:47.211585 19474 raft_consensus.cc:729] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfec9fef2a5747ef93d7f62f1cf07621, State: Initialized, Role: FOLLOWER
I20260927 11:16:47.211700 19474 consensus_queue.cc:260] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0569998203ec46bb924d6ddab403a0ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34503 } } peers { permanent_uuid: "cfec9fef2a5747ef93d7f62f1cf07621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44853 } } peers { permanent_uuid: "a8cf562ab87f41f288043da78629f273" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45863 } }
I20260927 11:16:47.211932 19477 tablet_bootstrap.cc:492] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273: No bootstrap required, opened a new log
I20260927 11:16:47.211943 19474 ts_tablet_manager.cc:1428] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:16:47.212003 19477 ts_tablet_manager.cc:1397] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:16:47.212021 19197 heartbeater.cc:503] Master 127.0.98.62:41419 was elected leader, sending a full tablet report...
I20260927 11:16:47.213909 19477 raft_consensus.cc:348] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0569998203ec46bb924d6ddab403a0ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34503 } } peers { permanent_uuid: "cfec9fef2a5747ef93d7f62f1cf07621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44853 } } peers { permanent_uuid: "a8cf562ab87f41f288043da78629f273" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45863 } }
I20260927 11:16:47.214025 19477 raft_consensus.cc:374] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:47.214046 19477 raft_consensus.cc:729] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8cf562ab87f41f288043da78629f273, State: Initialized, Role: FOLLOWER
I20260927 11:16:47.214206 19477 consensus_queue.cc:260] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0569998203ec46bb924d6ddab403a0ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34503 } } peers { permanent_uuid: "cfec9fef2a5747ef93d7f62f1cf07621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44853 } } peers { permanent_uuid: "a8cf562ab87f41f288043da78629f273" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45863 } }
I20260927 11:16:47.214391 19477 ts_tablet_manager.cc:1428] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 11:16:47.214461 19328 heartbeater.cc:503] Master 127.0.98.62:41419 was elected leader, sending a full tablet report...
I20260927 11:16:47.258003 19480 raft_consensus.cc:482] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:16:47.258108 19480 raft_consensus.cc:504] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0569998203ec46bb924d6ddab403a0ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34503 } } peers { permanent_uuid: "cfec9fef2a5747ef93d7f62f1cf07621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44853 } } peers { permanent_uuid: "a8cf562ab87f41f288043da78629f273" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45863 } }
I20260927 11:16:47.258373 19480 leader_election.cc:288] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0569998203ec46bb924d6ddab403a0ab (127.0.98.3:34503), a8cf562ab87f41f288043da78629f273 (127.0.98.2:45863)
I20260927 11:16:47.261037 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.258590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60367 (local address 127.0.98.2:45863)
0927 11:16:47.258649 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:47.258651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:47.258691 (+ 40us) server_negotiation.cc:407] Connection header received
0927 11:16:47.258778 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:47.258778 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:47.258788 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:47.258843 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:47.258982 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.259443 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.260346 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.260691 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.260781 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.260790 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:47.260938 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 11:16:47.260972 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 11:16:47.261040 19199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.258526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45863 (local address 127.0.98.1:60367)
0927 11:16:47.258647 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:47.258659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:47.258715 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:47.258845 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:47.258846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:47.258847 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:47.258907 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:47.258908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.259536 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.259537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:47.260258 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:47.260259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.260783 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.260784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:47.260852 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.260860 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 11:16:47.260931 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:16:47.260973 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":1}
I20260927 11:16:47.261242 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.258538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34503 (local address 127.0.98.1:49099)
0927 11:16:47.258713 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:47.258721 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:47.258796 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:47.258917 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:47.258918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:47.258920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:47.258988 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:47.258989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.259592 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.259593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:47.260323 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:47.260325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.260851 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.260852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:47.260936 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.260939 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:47.261007 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 11:16:47.261044 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":74,"threads_started":1}
I20260927 11:16:47.261364 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.258591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49099 (local address 127.0.98.3:34503)
0927 11:16:47.258655 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:47.258662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:47.258779 (+ 117us) server_negotiation.cc:407] Connection header received
0927 11:16:47.258850 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:47.258851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:47.258861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:47.258906 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:47.259055 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.259509 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.260420 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.260769 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.260849 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.260858 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:47.261262 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 11:16:47.261305 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 11:16:47.261440 19282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f258fef43344a680f8f7d598e0923a" candidate_uuid: "cfec9fef2a5747ef93d7f62f1cf07621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8cf562ab87f41f288043da78629f273" is_pre_election: true
I20260927 11:16:47.261580 19282 raft_consensus.cc:2454] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfec9fef2a5747ef93d7f62f1cf07621 in term 0.
I20260927 11:16:47.261596 19413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f258fef43344a680f8f7d598e0923a" candidate_uuid: "cfec9fef2a5747ef93d7f62f1cf07621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0569998203ec46bb924d6ddab403a0ab" is_pre_election: true
I20260927 11:16:47.261714 19413 raft_consensus.cc:2454] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cfec9fef2a5747ef93d7f62f1cf07621 in term 0.
I20260927 11:16:47.261795 19086 leader_election.cc:302] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8cf562ab87f41f288043da78629f273, cfec9fef2a5747ef93d7f62f1cf07621; no voters:
I20260927 11:16:47.261900 19480 raft_consensus.cc:2781] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:16:47.261948 19480 raft_consensus.cc:482] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:16:47.261969 19480 raft_consensus.cc:3037] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:47.262671 19480 raft_consensus.cc:504] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0569998203ec46bb924d6ddab403a0ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34503 } } peers { permanent_uuid: "cfec9fef2a5747ef93d7f62f1cf07621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44853 } } peers { permanent_uuid: "a8cf562ab87f41f288043da78629f273" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45863 } }
I20260927 11:16:47.262801 19480 leader_election.cc:288] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [CANDIDATE]: Term 1 election: Requested vote from peers 0569998203ec46bb924d6ddab403a0ab (127.0.98.3:34503), a8cf562ab87f41f288043da78629f273 (127.0.98.2:45863)
I20260927 11:16:47.262928 19413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f258fef43344a680f8f7d598e0923a" candidate_uuid: "cfec9fef2a5747ef93d7f62f1cf07621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0569998203ec46bb924d6ddab403a0ab"
I20260927 11:16:47.262941 19282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23f258fef43344a680f8f7d598e0923a" candidate_uuid: "cfec9fef2a5747ef93d7f62f1cf07621" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8cf562ab87f41f288043da78629f273"
I20260927 11:16:47.263017 19413 raft_consensus.cc:3037] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:47.263015 19282 raft_consensus.cc:3037] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:47.263666 19413 raft_consensus.cc:2454] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfec9fef2a5747ef93d7f62f1cf07621 in term 1.
I20260927 11:16:47.263856 19085 leader_election.cc:302] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0569998203ec46bb924d6ddab403a0ab, cfec9fef2a5747ef93d7f62f1cf07621; no voters:
I20260927 11:16:47.263890 19282 raft_consensus.cc:2454] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cfec9fef2a5747ef93d7f62f1cf07621 in term 1.
I20260927 11:16:47.263954 19480 raft_consensus.cc:2781] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:47.264097 19480 raft_consensus.cc:686] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [term 1 LEADER]: Becoming Leader. State: Replica: cfec9fef2a5747ef93d7f62f1cf07621, State: Running, Role: LEADER
I20260927 11:16:47.264232 19480 consensus_queue.cc:237] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0569998203ec46bb924d6ddab403a0ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34503 } } peers { permanent_uuid: "cfec9fef2a5747ef93d7f62f1cf07621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44853 } } peers { permanent_uuid: "a8cf562ab87f41f288043da78629f273" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45863 } }
I20260927 11:16:47.264954 19009 catalog_manager.cc:5495] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 reported cstate change: term changed from 0 to 1, leader changed from <none> to cfec9fef2a5747ef93d7f62f1cf07621 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cfec9fef2a5747ef93d7f62f1cf07621" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0569998203ec46bb924d6ddab403a0ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfec9fef2a5747ef93d7f62f1cf07621" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8cf562ab87f41f288043da78629f273" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45863 } health_report { overall_health: UNKNOWN } } }
I20260927 11:16:47.274876 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.272820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49522 (local address 127.0.98.1:44853)
0927 11:16:47.272862 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:47.272864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:47.272882 (+ 18us) server_negotiation.cc:407] Connection header received
0927 11:16:47.272942 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:47.272943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:47.272956 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:47.273008 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:16:47.273095 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.273492 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.274211 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:47.274389 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:47.274460 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:47.274565 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:16:47.274646 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:16:47.274709 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:47.274755 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 11:16:47.274763 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 11:16:47.276343 19413 raft_consensus.cc:1261] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab [term 1 FOLLOWER]: Refusing update from remote peer cfec9fef2a5747ef93d7f62f1cf07621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:16:47.276360 19282 raft_consensus.cc:1261] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273 [term 1 FOLLOWER]: Refusing update from remote peer cfec9fef2a5747ef93d7f62f1cf07621: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:16:47.276708 19480 consensus_queue.cc:1035] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0569998203ec46bb924d6ddab403a0ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:16:47.276870 19483 consensus_queue.cc:1035] T 23f258fef43344a680f8f7d598e0923a P cfec9fef2a5747ef93d7f62f1cf07621 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8cf562ab87f41f288043da78629f273" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:16:47.279381 19486 mvcc.cc:205] Tried to move back new op lower bound from 7333919978600767488 to 7333919978554015744. Current Snapshot: MvccSnapshot[applied={T|T < 7333919978600767488}]
I20260927 11:16:47.279698 19487 mvcc.cc:205] Tried to move back new op lower bound from 7333919978600767488 to 7333919978554015744. Current Snapshot: MvccSnapshot[applied={T|T < 7333919978600767488}]
I20260927 11:16:47.281445 19008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:16:47.281525 19008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:16:47.282130 19008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de8f830ca9bb4fc69eb5b61be343af2f: Sending DeleteTablet for 3 replicas of tablet 23f258fef43344a680f8f7d598e0923a
I20260927 11:16:47.282480 19393 tablet_service.cc:1514] Processing DeleteTablet for tablet 23f258fef43344a680f8f7d598e0923a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:47 UTC) from {username='slave'} at 127.0.0.1:51216
I20260927 11:16:47.282539 19131 tablet_service.cc:1514] Processing DeleteTablet for tablet 23f258fef43344a680f8f7d598e0923a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:47 UTC) from {username='slave'} at 127.0.0.1:49508
I20260927 11:16:47.282703 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 19069
I20260927 11:16:47.282773 19495 tablet_replica.cc:330] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab: stopping tablet replica
I20260927 11:16:47.282949 19262 tablet_service.cc:1514] Processing DeleteTablet for tablet 23f258fef43344a680f8f7d598e0923a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:47 UTC) from {username='slave'} at 127.0.0.1:49386
I20260927 11:16:47.284538 19495 raft_consensus.cc:2229] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:16:47.284662 19495 raft_consensus.cc:2258] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:47.285032 19495 ts_tablet_manager.cc:1905] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:16:47.286681 19495 ts_tablet_manager.cc:1918] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:16:47.286738 19495 log.cc:1196] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/23f258fef43344a680f8f7d598e0923a
I20260927 11:16:47.286966 19495 ts_tablet_manager.cc:1939] T 23f258fef43344a680f8f7d598e0923a P 0569998203ec46bb924d6ddab403a0ab: Deleting consensus metadata
I20260927 11:16:47.287333 18994 catalog_manager.cc:4841] TS 0569998203ec46bb924d6ddab403a0ab (127.0.98.3:34503): tablet 23f258fef43344a680f8f7d598e0923a (table test-table [id=ec109765fd82426dbf0ab65d1ee50c05]) successfully deleted
W20260927 11:16:47.288053 18996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44853 (error 108)
I20260927 11:16:47.288256 19497 tablet_replica.cc:330] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273: stopping tablet replica
I20260927 11:16:47.288352 19497 raft_consensus.cc:2229] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:16:47.288415 19497 raft_consensus.cc:2258] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:47.288507 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.288384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44853 (local address 127.0.0.1:49530)
0927 11:16:47.288435 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:47.288476 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 11:16:47.288611 18996 catalog_manager.cc:4584] TS cfec9fef2a5747ef93d7f62f1cf07621 (127.0.98.1:44853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 23f258fef43344a680f8f7d598e0923a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44853: connect: Connection refused (error 111)
I20260927 11:16:47.288640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 19200
I20260927 11:16:47.288841 19497 ts_tablet_manager.cc:1905] T 23f258fef43344a680f8f7d598e0923a P a8cf562ab87f41f288043da78629f273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:16:47.292179 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:47.292085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45863 (local address 127.0.0.1:49396)
0927 11:16:47.292133 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:47.292156 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 11:16:47.292179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 19331
I20260927 11:16:47.298141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 18978
2026-09-27T11:16:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (696 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:16:47 dist-test-slave-9hq9 krb5kdc[19501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:16:47 dist-test-slave-9hq9 krb5kdc[19501](info): set up 2 sockets
Sep 27 11:16:47 dist-test-slave-9hq9 krb5kdc[19501](info): commencing operation
krb5kdc: starting...
W20260927 11:16:48.313827 392 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:16:48 dist-test-slave-9hq9 krb5kdc[19501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:16:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:16:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:16:48.424417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38157
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:48.501940 19514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:48.502189 19514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:48.502228 19514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:48.503382 19514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:48.503450 19514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:48.503476 19514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:48.503508 19514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:48.503532 19514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:48.505103 19514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:48.505374 19514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:48.505609 19514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:48.510329 19519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:48.510341 19520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:48.510334 19514 server_base.cc:1034] running on GCE node
W20260927 11:16:48.510578 19522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:48.510819 19514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:48.511044 19514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:48.512210 19514 hybrid_clock.cc:648] HybridClock initialized: now 1790507808512195 us; error 31 us; skew 500 ppm
Sep 27 11:16:48 dist-test-slave-9hq9 krb5kdc[19501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:48.514529 19514 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:16:48.514925 19514 webserver.cc:460] Webserver started at http://127.0.98.62:42943/ using document root <none> and password file <none>
I20260927 11:16:48.515130 19514 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:48.515193 19514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:48.515339 19514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:48.516309 19514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "4933409a8d6b4e29b51f884e9d1e1a77"
format_stamp: "Formatted at 2026-09-27 11:16:48 on dist-test-slave-9hq9"
server_key: "98f3b8ef151d37768fca4d36e488dd24"
server_key_iv: "f2a46e0076c33d14a1c4c54427399bed"
server_key_version: "encryptionkey@0"
I20260927 11:16:48.516649 19514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "4933409a8d6b4e29b51f884e9d1e1a77"
format_stamp: "Formatted at 2026-09-27 11:16:48 on dist-test-slave-9hq9"
server_key: "98f3b8ef151d37768fca4d36e488dd24"
server_key_iv: "f2a46e0076c33d14a1c4c54427399bed"
server_key_version: "encryptionkey@0"
I20260927 11:16:48.517871 19514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:16:48.518638 19529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:48.518796 19514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:48.518891 19514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "4933409a8d6b4e29b51f884e9d1e1a77"
format_stamp: "Formatted at 2026-09-27 11:16:48 on dist-test-slave-9hq9"
server_key: "98f3b8ef151d37768fca4d36e488dd24"
server_key_iv: "f2a46e0076c33d14a1c4c54427399bed"
server_key_version: "encryptionkey@0"
I20260927 11:16:48.518977 19514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:48.532992 19514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:48.534669 19514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:48.534819 19514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:48.538401 19514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38157
I20260927 11:16:48.538467 19581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38157 every 8 connection(s)
I20260927 11:16:48.538838 19514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:48.539644 19582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:48.542253 19582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77: Bootstrap starting.
I20260927 11:16:48.542799 19582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:48.543085 19582 log.cc:824] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:48.543713 19582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77: No bootstrap required, opened a new log
I20260927 11:16:48.545174 19582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4933409a8d6b4e29b51f884e9d1e1a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38157 } }
I20260927 11:16:48.545295 19582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:48.545353 19582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4933409a8d6b4e29b51f884e9d1e1a77, State: Initialized, Role: FOLLOWER
I20260927 11:16:48.545511 19582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4933409a8d6b4e29b51f884e9d1e1a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38157 } }
I20260927 11:16:48.545589 19582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:48.545637 19582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:48.545684 19582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:48.546235 19582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4933409a8d6b4e29b51f884e9d1e1a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38157 } }
I20260927 11:16:48.546335 19582 leader_election.cc:302] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4933409a8d6b4e29b51f884e9d1e1a77; no voters:
I20260927 11:16:48.546485 19582 leader_election.cc:288] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:16:48.546501 19585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:48.546684 19585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [term 1 LEADER]: Becoming Leader. State: Replica: 4933409a8d6b4e29b51f884e9d1e1a77, State: Running, Role: LEADER
I20260927 11:16:48.546733 19582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:48.546855 19585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4933409a8d6b4e29b51f884e9d1e1a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38157 } }
I20260927 11:16:48.547418 19587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4933409a8d6b4e29b51f884e9d1e1a77. Latest consensus state: current_term: 1 leader_uuid: "4933409a8d6b4e29b51f884e9d1e1a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4933409a8d6b4e29b51f884e9d1e1a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38157 } } }
I20260927 11:16:48.547515 19587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:48.547631 19586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4933409a8d6b4e29b51f884e9d1e1a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4933409a8d6b4e29b51f884e9d1e1a77" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38157 } } }
I20260927 11:16:48.547735 19594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:48.547744 19586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:48.548425 19594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:16:48.548923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 19514
I20260927 11:16:48.549036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:16:48.549218 392 external_mini_cluster.cc:1428] Setting key b2d992c53f371d5ca5e0671ccea2f70e
I20260927 11:16:48.549913 19594 catalog_manager.cc:1347] Generated new cluster ID: afc1582f20bb4d22a7d6843c0719ae4d
I20260927 11:16:48.549961 19594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:16:48 dist-test-slave-9hq9 krb5kdc[19501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:16:48.566617 19594 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:16:48.567271 19594 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:48.568849 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:48.549776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.0.98.62:38157)
0927 11:16:48.550137 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:48.550146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:48.550178 (+ 32us) server_negotiation.cc:407] Connection header received
0927 11:16:48.558301 (+ 8123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:48.558309 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:48.558368 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:48.558513 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:48.559196 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:48.560618 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:48.561620 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:48.562277 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:48.562401 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:48.565080 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:48.565084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:48.565085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:48.565095 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:48.567204 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:48.567590 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:48.567593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:48.567593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:48.567689 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:48.567901 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:48.567902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:48.567902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:48.568141 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:48.568304 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:48.568387 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 11:16:48.568515 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":71,"threads_started":1}
I20260927 11:16:48.575466 19594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77: Generated new TSK 0
I20260927 11:16:48.575629 19594 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 11:16:48.602492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38157
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:48.698201 19608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:48.698381 19608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:48.698403 19608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:48.699466 19608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:48.699497 19608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:48.699522 19608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:16:48.700894 19608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:48.701071 19608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:48.701227 19608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:48.706073 19616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:48.706069 19614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:48.706095 19613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:48.706204 19608 server_base.cc:1034] running on GCE node
I20260927 11:16:48.706624 19608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:48.706851 19608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:48.708000 19608 hybrid_clock.cc:648] HybridClock initialized: now 1790507808707984 us; error 33 us; skew 500 ppm
Sep 27 11:16:48 dist-test-slave-9hq9 krb5kdc[19501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:48.710263 19608 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 11:16:48.710649 19608 webserver.cc:460] Webserver started at http://127.0.98.1:45313/ using document root <none> and password file <none>
I20260927 11:16:48.710872 19608 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:48.710928 19608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:48.711069 19608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:48.712054 19608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "ec89a8387ff2463a831f76624d0fbe52"
format_stamp: "Formatted at 2026-09-27 11:16:48 on dist-test-slave-9hq9"
server_key: "458c249e4a35fa9116717f9f75217e5c"
server_key_iv: "7e28b4422a88d83b3e1db61c544b00df"
server_key_version: "encryptionkey@0"
I20260927 11:16:48.712383 19608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ec89a8387ff2463a831f76624d0fbe52"
format_stamp: "Formatted at 2026-09-27 11:16:48 on dist-test-slave-9hq9"
server_key: "458c249e4a35fa9116717f9f75217e5c"
server_key_iv: "7e28b4422a88d83b3e1db61c544b00df"
server_key_version: "encryptionkey@0"
I20260927 11:16:48.713599 19608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:16:48.714306 19623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:48.714463 19608 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:16:48.714557 19608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "ec89a8387ff2463a831f76624d0fbe52"
format_stamp: "Formatted at 2026-09-27 11:16:48 on dist-test-slave-9hq9"
server_key: "458c249e4a35fa9116717f9f75217e5c"
server_key_iv: "7e28b4422a88d83b3e1db61c544b00df"
server_key_version: "encryptionkey@0"
I20260927 11:16:48.714628 19608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:48.721029 19608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:48.722844 19608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:48.722987 19608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:48.723177 19608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:48.723449 19608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:48.723500 19608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:48.723537 19608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:48.723563 19608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:48.729208 19608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40139
I20260927 11:16:48.729254 19736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40139 every 8 connection(s)
I20260927 11:16:48.729517 19608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 11:16:48 dist-test-slave-9hq9 krb5kdc[19501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:16:48.737212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 19608
I20260927 11:16:48.737332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:16:48.737537 392 external_mini_cluster.cc:1428] Setting key 6fa60eb4601fd0bb3c5b55b55f0b5476
I20260927 11:16:48.738238 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:48.730156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49605 (local address 127.0.98.62:38157)
0927 11:16:48.730213 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:48.730216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:48.730385 (+ 169us) server_negotiation.cc:407] Connection header received
0927 11:16:48.731252 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:48.731253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:48.731265 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:48.731315 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:48.731910 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:48.732410 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:48.733405 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:48.733610 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:48.733688 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:48.735550 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:48.735551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:48.735552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:48.735559 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:48.736699 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:48.737122 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:48.737124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:48.737124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:48.737182 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:48.737484 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:48.737485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:48.737485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:48.737592 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:48.737688 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:48.738022 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 11:16:48.738142 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:16:48.738394 19739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:48.730122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38157 (local address 127.0.98.1:49605)
0927 11:16:48.730271 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:48.730312 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:48.731141 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:48.731373 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:48.731376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:48.731599 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:48.731814 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:48.731818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:48.732508 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:48.732509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:48.733302 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:48.733304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:48.733699 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:48.733700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:48.733824 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:48.734264 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:48.734272 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:48.735459 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:48.736792 (+ 1333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:48.736799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:48.736801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:48.737022 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:48.737246 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:48.737246 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:48.737247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:48.737377 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:48.737713 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:48.737721 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:48.737881 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 11:16:48.737981 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 11:16:48.738157 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260927 11:16:48.738828 19737 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38157
I20260927 11:16:48.738932 19737 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:48.739090 19737 heartbeater.cc:511] Master 127.0.98.62:38157 requested a full tablet report, sending...
I20260927 11:16:48.739650 19546 ts_manager.cc:194] Registered new tserver with Master: ec89a8387ff2463a831f76624d0fbe52 (127.0.98.1:40139)
I20260927 11:16:48.740547 19546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49605
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 11:16:48.761292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38157
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:48.838687 19742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:48.838865 19742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:48.838887 19742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:48.840003 19742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:48.840040 19742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:48.840090 19742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:16:48.841531 19742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:48.841753 19742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:48.841969 19742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:48.846653 19748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:48.846689 19750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:48.846748 19742 server_base.cc:1034] running on GCE node
W20260927 11:16:48.846700 19747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:48.847036 19742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:48.847209 19742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:48.848337 19742 hybrid_clock.cc:648] HybridClock initialized: now 1790507808848326 us; error 26 us; skew 500 ppm
Sep 27 11:16:48 dist-test-slave-9hq9 krb5kdc[19501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:48.850616 19742 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 11:16:48.851039 19742 webserver.cc:460] Webserver started at http://127.0.98.2:43199/ using document root <none> and password file <none>
I20260927 11:16:48.851250 19742 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:48.851290 19742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:48.851442 19742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:48.852397 19742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "ca4d9b3986f6496eb76572cae18044fc"
format_stamp: "Formatted at 2026-09-27 11:16:48 on dist-test-slave-9hq9"
server_key: "e1169ca409432b4dc350df20dcba0085"
server_key_iv: "cbeb60eb462a7bcfbbb9742d7b871e10"
server_key_version: "encryptionkey@0"
I20260927 11:16:48.852727 19742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ca4d9b3986f6496eb76572cae18044fc"
format_stamp: "Formatted at 2026-09-27 11:16:48 on dist-test-slave-9hq9"
server_key: "e1169ca409432b4dc350df20dcba0085"
server_key_iv: "cbeb60eb462a7bcfbbb9742d7b871e10"
server_key_version: "encryptionkey@0"
I20260927 11:16:48.853919 19742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:16:48.854628 19757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:48.854766 19742 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:48.854851 19742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "ca4d9b3986f6496eb76572cae18044fc"
format_stamp: "Formatted at 2026-09-27 11:16:48 on dist-test-slave-9hq9"
server_key: "e1169ca409432b4dc350df20dcba0085"
server_key_iv: "cbeb60eb462a7bcfbbb9742d7b871e10"
server_key_version: "encryptionkey@0"
I20260927 11:16:48.854923 19742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:48.871309 19742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:48.873016 19742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:48.873167 19742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:48.873364 19742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:48.873637 19742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:48.873687 19742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:48.873724 19742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:48.873749 19742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:48.879518 19742 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32891
I20260927 11:16:48.879563 19870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32891 every 8 connection(s)
I20260927 11:16:48.879846 19742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 11:16:48 dist-test-slave-9hq9 krb5kdc[19501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:16:48.886011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 19742
I20260927 11:16:48.886147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:16:48.886379 392 external_mini_cluster.cc:1428] Setting key cb3cb68e23690167e97af50af6902aaf
I20260927 11:16:48.888221 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:48.880407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38157 (local address 127.0.98.2:39973)
0927 11:16:48.880560 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:48.880596 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:48.881434 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:48.881650 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:48.881653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:48.881870 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:48.882028 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:48.882032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:48.882683 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:48.882683 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:48.883411 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:48.883414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:48.883703 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:48.883703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:48.883790 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:48.884174 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:48.884181 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:48.885423 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:48.886983 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:48.886989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:48.886991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:48.887208 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:48.887417 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:48.887418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:48.887418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:48.887516 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:48.887784 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:48.887788 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:48.887899 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 11:16:48.887975 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 11:16:48.888079 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260927 11:16:48.888242 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:48.880448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39973 (local address 127.0.98.62:38157)
0927 11:16:48.880492 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:48.880494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:48.880685 (+ 191us) server_negotiation.cc:407] Connection header received
0927 11:16:48.881515 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:48.881516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:48.881524 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:48.881576 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:48.882165 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:48.882614 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:48.883481 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:48.883642 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:48.883685 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:48.885505 (+ 1820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:48.885506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:48.885506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:48.885513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:48.886911 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:48.887288 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:48.887289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:48.887290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:48.887348 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:48.887575 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:48.887575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:48.887576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:48.887705 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:48.887791 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:48.888018 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 11:16:48.888102 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:16:48.888834 19871 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38157
I20260927 11:16:48.888934 19871 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:48.889114 19871 heartbeater.cc:511] Master 127.0.98.62:38157 requested a full tablet report, sending...
I20260927 11:16:48.889434 19546 ts_manager.cc:194] Registered new tserver with Master: ca4d9b3986f6496eb76572cae18044fc (127.0.98.2:32891)
I20260927 11:16:48.889796 19546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39973
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 11:16:48.910655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38157
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:48.987304 19876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:48.987527 19876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:48.987565 19876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:48.988684 19876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:48.988750 19876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:48.988793 19876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:16:48.990232 19876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:48.990473 19876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:48.990680 19876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:48.995460 19884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:48.995442 19876 server_base.cc:1034] running on GCE node
W20260927 11:16:48.995442 19881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:48.995434 19882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:48.995844 19876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:48.996104 19876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:48.997258 19876 hybrid_clock.cc:648] HybridClock initialized: now 1790507808997246 us; error 42 us; skew 500 ppm
Sep 27 11:16:48 dist-test-slave-9hq9 krb5kdc[19501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:48.999610 19876 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 11:16:48.999996 19876 webserver.cc:460] Webserver started at http://127.0.98.3:39757/ using document root <none> and password file <none>
I20260927 11:16:49.000205 19876 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:49.000268 19876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:49.000408 19876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:49.001341 19876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "76dd51fd5aee43b5ac424ed56eb89d0d"
format_stamp: "Formatted at 2026-09-27 11:16:48 on dist-test-slave-9hq9"
server_key: "6600c5e4dc739534acc59e4d35b891d3"
server_key_iv: "51ff9c4168537f3f4f3a0ccfcd3c2e53"
server_key_version: "encryptionkey@0"
I20260927 11:16:49.001704 19876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "76dd51fd5aee43b5ac424ed56eb89d0d"
format_stamp: "Formatted at 2026-09-27 11:16:48 on dist-test-slave-9hq9"
server_key: "6600c5e4dc739534acc59e4d35b891d3"
server_key_iv: "51ff9c4168537f3f4f3a0ccfcd3c2e53"
server_key_version: "encryptionkey@0"
I20260927 11:16:49.002978 19876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:16:49.003644 19891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:49.003803 19876 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:16:49.003887 19876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "76dd51fd5aee43b5ac424ed56eb89d0d"
format_stamp: "Formatted at 2026-09-27 11:16:48 on dist-test-slave-9hq9"
server_key: "6600c5e4dc739534acc59e4d35b891d3"
server_key_iv: "51ff9c4168537f3f4f3a0ccfcd3c2e53"
server_key_version: "encryptionkey@0"
I20260927 11:16:49.003961 19876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:49.012678 19876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:49.014775 19876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:49.014921 19876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:49.015120 19876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:49.015374 19876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:49.015424 19876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:49.015460 19876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:49.015487 19876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:49.021148 19876 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45501
I20260927 11:16:49.021195 20004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45501 every 8 connection(s)
I20260927 11:16:49.021497 19876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:16:49.025132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 19876
I20260927 11:16:49.025303 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:16:49.025477 392 external_mini_cluster.cc:1428] Setting key 4c2aefcef659bf1e86efb4671f92bbf9
Sep 27 11:16:49 dist-test-slave-9hq9 krb5kdc[19501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507808, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:16:49.029788 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.022112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43665 (local address 127.0.98.62:38157)
0927 11:16:49.022192 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.022194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.022354 (+ 160us) server_negotiation.cc:407] Connection header received
0927 11:16:49.023155 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.023156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.023165 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.023212 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:49.023742 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.024227 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.025162 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.025485 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.025576 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.027527 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:49.027528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:49.027528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:49.027534 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:49.028690 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:49.029033 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:49.029034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:49.029034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:49.029082 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:49.029285 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:49.029286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:49.029286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:49.029378 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:49.029426 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.029648 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.029728 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:16:49.029912 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.022035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38157 (local address 127.0.98.3:43665)
0927 11:16:49.022250 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.022279 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:49.023076 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:49.023257 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:49.023259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:49.023463 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:49.023667 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.023671 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.024299 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.024300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.025046 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.025048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.025578 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.025578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.025718 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.026205 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:16:49.026216 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:49.027441 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:49.028757 (+ 1316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:49.028762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:49.028764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:49.028965 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:49.029136 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:16:49.029137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:16:49.029137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:16:49.029225 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:16:49.029428 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:49.029431 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:49.029531 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 11:16:49.029622 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 11:16:49.029740 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260927 11:16:49.030402 20005 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38157
I20260927 11:16:49.030498 20005 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:49.030659 20005 heartbeater.cc:511] Master 127.0.98.62:38157 requested a full tablet report, sending...
I20260927 11:16:49.030938 19546 ts_manager.cc:194] Registered new tserver with Master: 76dd51fd5aee43b5ac424ed56eb89d0d (127.0.98.3:45501)
I20260927 11:16:49.031275 19546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43665
I20260927 11:16:49.037022 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:16:49.043475 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.038355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.0.98.62:38157)
0927 11:16:49.038404 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.038405 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.038513 (+ 108us) server_negotiation.cc:407] Connection header received
0927 11:16:49.038551 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.038552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.038560 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.038614 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:49.039126 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.039538 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.040177 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.040340 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.040384 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.041445 (+ 1061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:49.041445 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:49.041446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:49.041450 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:49.042608 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:49.042877 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:49.042877 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:49.042877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:49.042920 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:49.043058 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:49.043058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:49.043058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:49.043168 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:49.043233 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.043376 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.043425 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 11:16:49.045058 19546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:16:49.046639 19546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:16:49.052474 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.048636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45501 (local address 127.0.0.1:34682)
0927 11:16:49.048880 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.048889 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:49.048929 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:49.049269 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:49.049270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:49.049272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:49.049301 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.049302 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.050450 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.050451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.051455 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.051458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.052278 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.052279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.052331 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.052340 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 11:16:49.052393 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 11:16:49.052426 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":83,"thread_start_us":56,"threads_started":1}
I20260927 11:16:49.052536 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.048745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34682 (local address 127.0.98.3:45501)
0927 11:16:49.049052 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.049057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.049077 (+ 20us) server_negotiation.cc:407] Connection header received
0927 11:16:49.049104 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.049108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.049121 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.049189 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:49.049364 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.050330 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.051536 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.052198 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.052271 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.052320 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.052427 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.052462 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":76,"threads_started":1}
I20260927 11:16:49.052837 20013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.048766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32891 (local address 127.0.0.1:37584)
0927 11:16:49.049250 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.049259 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:49.049335 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:49.049644 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:49.049645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:49.049646 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:49.049684 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.049685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.050614 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.050614 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.051720 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.051723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.052610 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.052611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.052690 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.052694 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:16:49.052741 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 11:16:49.052779 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":4,"thread_start_us":35,"threads_started":1}
I20260927 11:16:49.053113 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.049041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37584 (local address 127.0.98.2:32891)
0927 11:16:49.049240 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.049243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.049324 (+ 81us) server_negotiation.cc:407] Connection header received
0927 11:16:49.049395 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.049397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.049410 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.049506 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:49.049759 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.050509 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.051801 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.052536 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.052612 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.052669 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.052982 (+ 313us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.053032 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":106,"threads_started":1}
I20260927 11:16:49.053284 19939 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8ffc54f21b4bc99673ed5ccb769c2e (DEFAULT_TABLE table=test-table [id=404577447fd9458ab75e7ccfb4ef7dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:49.053578 19939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8ffc54f21b4bc99673ed5ccb769c2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:49.053712 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.048688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49990 (local address 127.0.98.1:40139)
0927 11:16:49.048875 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.048879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.048890 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:16:49.048927 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.048929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.048945 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.049037 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:49.049214 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.050336 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.052702 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.053337 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.053398 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.053448 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.053611 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.053648 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
I20260927 11:16:49.053726 19805 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8ffc54f21b4bc99673ed5ccb769c2e (DEFAULT_TABLE table=test-table [id=404577447fd9458ab75e7ccfb4ef7dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:49.053874 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.048577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40139 (local address 127.0.0.1:49990)
0927 11:16:49.048735 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.048771 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:49.048837 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:49.049049 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:49.049052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:49.049054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:49.049123 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.049125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.050459 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.050460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.051720 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.051723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.053474 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.053475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.053524 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.053527 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:49.053590 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 11:16:49.053626 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":87,"thread_start_us":43,"threads_started":1}
I20260927 11:16:49.053983 19805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8ffc54f21b4bc99673ed5ccb769c2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:49.054354 19671 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8ffc54f21b4bc99673ed5ccb769c2e (DEFAULT_TABLE table=test-table [id=404577447fd9458ab75e7ccfb4ef7dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:49.054557 19671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8ffc54f21b4bc99673ed5ccb769c2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:49.056902 20019 tablet_bootstrap.cc:492] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d: Bootstrap starting.
I20260927 11:16:49.057072 20021 tablet_bootstrap.cc:492] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc: Bootstrap starting.
I20260927 11:16:49.057689 20021 tablet_bootstrap.cc:654] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:49.057689 20019 tablet_bootstrap.cc:654] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:49.058215 20020 tablet_bootstrap.cc:492] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52: Bootstrap starting.
I20260927 11:16:49.058336 20021 log.cc:824] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:49.058357 20019 log.cc:824] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:49.058880 20020 tablet_bootstrap.cc:654] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:49.059031 20019 tablet_bootstrap.cc:492] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d: No bootstrap required, opened a new log
I20260927 11:16:49.059126 20019 ts_tablet_manager.cc:1397] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:16:49.059397 20020 log.cc:824] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:49.059440 20021 tablet_bootstrap.cc:492] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc: No bootstrap required, opened a new log
I20260927 11:16:49.059490 20021 ts_tablet_manager.cc:1397] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:16:49.059983 20020 tablet_bootstrap.cc:492] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52: No bootstrap required, opened a new log
I20260927 11:16:49.060076 20020 ts_tablet_manager.cc:1397] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:16:49.061028 20019 raft_consensus.cc:348] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec89a8387ff2463a831f76624d0fbe52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40139 } } peers { permanent_uuid: "ca4d9b3986f6496eb76572cae18044fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32891 } } peers { permanent_uuid: "76dd51fd5aee43b5ac424ed56eb89d0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45501 } }
I20260927 11:16:49.061198 20019 raft_consensus.cc:374] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:49.061236 20019 raft_consensus.cc:729] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76dd51fd5aee43b5ac424ed56eb89d0d, State: Initialized, Role: FOLLOWER
I20260927 11:16:49.061367 20019 consensus_queue.cc:260] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec89a8387ff2463a831f76624d0fbe52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40139 } } peers { permanent_uuid: "ca4d9b3986f6496eb76572cae18044fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32891 } } peers { permanent_uuid: "76dd51fd5aee43b5ac424ed56eb89d0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45501 } }
I20260927 11:16:49.061347 20021 raft_consensus.cc:348] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec89a8387ff2463a831f76624d0fbe52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40139 } } peers { permanent_uuid: "ca4d9b3986f6496eb76572cae18044fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32891 } } peers { permanent_uuid: "76dd51fd5aee43b5ac424ed56eb89d0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45501 } }
I20260927 11:16:49.061501 20021 raft_consensus.cc:374] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:49.061540 20021 raft_consensus.cc:729] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4d9b3986f6496eb76572cae18044fc, State: Initialized, Role: FOLLOWER
I20260927 11:16:49.061638 20019 ts_tablet_manager.cc:1428] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:16:49.061538 20020 raft_consensus.cc:348] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec89a8387ff2463a831f76624d0fbe52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40139 } } peers { permanent_uuid: "ca4d9b3986f6496eb76572cae18044fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32891 } } peers { permanent_uuid: "76dd51fd5aee43b5ac424ed56eb89d0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45501 } }
I20260927 11:16:49.061652 20021 consensus_queue.cc:260] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec89a8387ff2463a831f76624d0fbe52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40139 } } peers { permanent_uuid: "ca4d9b3986f6496eb76572cae18044fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32891 } } peers { permanent_uuid: "76dd51fd5aee43b5ac424ed56eb89d0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45501 } }
I20260927 11:16:49.061691 20020 raft_consensus.cc:374] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:49.061733 20020 raft_consensus.cc:729] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec89a8387ff2463a831f76624d0fbe52, State: Initialized, Role: FOLLOWER
I20260927 11:16:49.061764 20005 heartbeater.cc:503] Master 127.0.98.62:38157 was elected leader, sending a full tablet report...
I20260927 11:16:49.061833 20020 consensus_queue.cc:260] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec89a8387ff2463a831f76624d0fbe52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40139 } } peers { permanent_uuid: "ca4d9b3986f6496eb76572cae18044fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32891 } } peers { permanent_uuid: "76dd51fd5aee43b5ac424ed56eb89d0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45501 } }
I20260927 11:16:49.061918 19871 heartbeater.cc:503] Master 127.0.98.62:38157 was elected leader, sending a full tablet report...
I20260927 11:16:49.061918 20021 ts_tablet_manager.cc:1428] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 11:16:49.062124 20020 ts_tablet_manager.cc:1428] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:16:49.062143 19737 heartbeater.cc:503] Master 127.0.98.62:38157 was elected leader, sending a full tablet report...
I20260927 11:16:49.083164 20027 raft_consensus.cc:482] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:16:49.083228 20027 raft_consensus.cc:504] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec89a8387ff2463a831f76624d0fbe52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40139 } } peers { permanent_uuid: "ca4d9b3986f6496eb76572cae18044fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32891 } } peers { permanent_uuid: "76dd51fd5aee43b5ac424ed56eb89d0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45501 } }
I20260927 11:16:49.083477 20027 leader_election.cc:288] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca4d9b3986f6496eb76572cae18044fc (127.0.98.2:32891), 76dd51fd5aee43b5ac424ed56eb89d0d (127.0.98.3:45501)
I20260927 11:16:49.085915 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.083648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54639 (local address 127.0.98.2:32891)
0927 11:16:49.083711 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.083713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.083731 (+ 18us) server_negotiation.cc:407] Connection header received
0927 11:16:49.083751 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.083752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.083762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.083809 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:49.083966 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.084420 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.085267 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.085593 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.085676 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.085684 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.085831 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.085863 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:16:49.085915 19739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.083582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32891 (local address 127.0.98.1:54639)
0927 11:16:49.083634 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.083650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:49.083706 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:49.083827 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:49.083828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:49.083830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:49.083887 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.083888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.084493 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.084494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.085177 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.085178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.085678 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.085679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.085753 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.085755 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:16:49.085814 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 11:16:49.085852 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 11:16:49.086097 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.083725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45341 (local address 127.0.98.3:45501)
0927 11:16:49.083759 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.083761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.083841 (+ 80us) server_negotiation.cc:407] Connection header received
0927 11:16:49.083934 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.083935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.083944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.083972 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:49.084172 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.084568 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.085424 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.085756 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.085832 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.085839 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.085986 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.086016 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:16:49.086133 20028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.083626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45501 (local address 127.0.98.1:45341)
0927 11:16:49.083791 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.083800 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:49.083863 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:49.084030 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:49.084030 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:49.084032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:49.084103 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.084104 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.084653 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.084654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.085340 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.085341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.085836 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.085836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.085913 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.085916 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:49.085980 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 11:16:49.086015 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
I20260927 11:16:49.086216 19825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ffc54f21b4bc99673ed5ccb769c2e" candidate_uuid: "ec89a8387ff2463a831f76624d0fbe52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4d9b3986f6496eb76572cae18044fc" is_pre_election: true
I20260927 11:16:49.086351 19825 raft_consensus.cc:2454] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec89a8387ff2463a831f76624d0fbe52 in term 0.
I20260927 11:16:49.086422 19959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ffc54f21b4bc99673ed5ccb769c2e" candidate_uuid: "ec89a8387ff2463a831f76624d0fbe52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76dd51fd5aee43b5ac424ed56eb89d0d" is_pre_election: true
I20260927 11:16:49.086539 19627 leader_election.cc:302] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca4d9b3986f6496eb76572cae18044fc, ec89a8387ff2463a831f76624d0fbe52; no voters:
I20260927 11:16:49.086560 19959 raft_consensus.cc:2454] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec89a8387ff2463a831f76624d0fbe52 in term 0.
I20260927 11:16:49.086670 20027 raft_consensus.cc:2781] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:16:49.086724 20027 raft_consensus.cc:482] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:16:49.086750 20027 raft_consensus.cc:3037] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:49.087277 20027 raft_consensus.cc:504] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec89a8387ff2463a831f76624d0fbe52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40139 } } peers { permanent_uuid: "ca4d9b3986f6496eb76572cae18044fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32891 } } peers { permanent_uuid: "76dd51fd5aee43b5ac424ed56eb89d0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45501 } }
I20260927 11:16:49.087399 20027 leader_election.cc:288] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [CANDIDATE]: Term 1 election: Requested vote from peers ca4d9b3986f6496eb76572cae18044fc (127.0.98.2:32891), 76dd51fd5aee43b5ac424ed56eb89d0d (127.0.98.3:45501)
I20260927 11:16:49.087548 19959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ffc54f21b4bc99673ed5ccb769c2e" candidate_uuid: "ec89a8387ff2463a831f76624d0fbe52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76dd51fd5aee43b5ac424ed56eb89d0d"
I20260927 11:16:49.087533 19825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ffc54f21b4bc99673ed5ccb769c2e" candidate_uuid: "ec89a8387ff2463a831f76624d0fbe52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4d9b3986f6496eb76572cae18044fc"
I20260927 11:16:49.087610 19825 raft_consensus.cc:3037] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:49.087610 19959 raft_consensus.cc:3037] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:49.088351 19959 raft_consensus.cc:2454] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec89a8387ff2463a831f76624d0fbe52 in term 1.
I20260927 11:16:49.088351 19825 raft_consensus.cc:2454] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec89a8387ff2463a831f76624d0fbe52 in term 1.
I20260927 11:16:49.088497 19627 leader_election.cc:302] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca4d9b3986f6496eb76572cae18044fc, ec89a8387ff2463a831f76624d0fbe52; no voters:
I20260927 11:16:49.088591 20027 raft_consensus.cc:2781] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:49.088706 20027 raft_consensus.cc:686] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [term 1 LEADER]: Becoming Leader. State: Replica: ec89a8387ff2463a831f76624d0fbe52, State: Running, Role: LEADER
I20260927 11:16:49.088804 20027 consensus_queue.cc:237] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec89a8387ff2463a831f76624d0fbe52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40139 } } peers { permanent_uuid: "ca4d9b3986f6496eb76572cae18044fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32891 } } peers { permanent_uuid: "76dd51fd5aee43b5ac424ed56eb89d0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45501 } }
I20260927 11:16:49.089427 19545 catalog_manager.cc:5495] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec89a8387ff2463a831f76624d0fbe52 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ec89a8387ff2463a831f76624d0fbe52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec89a8387ff2463a831f76624d0fbe52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca4d9b3986f6496eb76572cae18044fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76dd51fd5aee43b5ac424ed56eb89d0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45501 } health_report { overall_health: UNKNOWN } } }
I20260927 11:16:49.097198 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.095235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.0.98.1:40139)
0927 11:16:49.095265 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.095267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.095298 (+ 31us) server_negotiation.cc:407] Connection header received
0927 11:16:49.095377 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.095378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.095390 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.095442 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:16:49.095519 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.095920 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.096603 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.096775 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.096820 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.096924 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:16:49.096993 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:16:49.097041 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.097090 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.097104 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:16:49.098551 19959 raft_consensus.cc:1261] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d [term 1 FOLLOWER]: Refusing update from remote peer ec89a8387ff2463a831f76624d0fbe52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:16:49.098558 19825 raft_consensus.cc:1261] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc [term 1 FOLLOWER]: Refusing update from remote peer ec89a8387ff2463a831f76624d0fbe52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:16:49.098814 20027 consensus_queue.cc:1035] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76dd51fd5aee43b5ac424ed56eb89d0d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:16:49.098924 20029 consensus_queue.cc:1035] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca4d9b3986f6496eb76572cae18044fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:16:49.101279 20032 mvcc.cc:205] Tried to move back new op lower bound from 7333919986064969728 to 7333919986027401216. Current Snapshot: MvccSnapshot[applied={T|T < 7333919986064969728}]
I20260927 11:16:49.101568 20033 mvcc.cc:205] Tried to move back new op lower bound from 7333919986064969728 to 7333919986027401216. Current Snapshot: MvccSnapshot[applied={T|T < 7333919986064969728}]
I20260927 11:16:49.102308 20035 mvcc.cc:205] Tried to move back new op lower bound from 7333919986064969728 to 7333919986027401216. Current Snapshot: MvccSnapshot[applied={T|T < 7333919986064969728}]
I20260927 11:16:49.103199 19546 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:16:49.103261 19546 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:16:49.103834 19546 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4933409a8d6b4e29b51f884e9d1e1a77: Sending DeleteTablet for 3 replicas of tablet 1c8ffc54f21b4bc99673ed5ccb769c2e
I20260927 11:16:49.104152 19939 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8ffc54f21b4bc99673ed5ccb769c2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34682
I20260927 11:16:49.104219 19671 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8ffc54f21b4bc99673ed5ccb769c2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49990
I20260927 11:16:49.104220 19805 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8ffc54f21b4bc99673ed5ccb769c2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37584
I20260927 11:16:49.104431 20042 tablet_replica.cc:330] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52: stopping tablet replica
I20260927 11:16:49.104537 20042 raft_consensus.cc:2229] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:16:49.104588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 19608
I20260927 11:16:49.104624 20042 raft_consensus.cc:2258] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ec89a8387ff2463a831f76624d0fbe52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:49.105178 20043 tablet_replica.cc:330] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc: stopping tablet replica
I20260927 11:16:49.105283 20043 raft_consensus.cc:2229] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:16:49.105376 20043 raft_consensus.cc:2258] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:49.105540 20041 tablet_replica.cc:330] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d: stopping tablet replica
I20260927 11:16:49.105639 20041 raft_consensus.cc:2229] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:16:49.105733 20041 raft_consensus.cc:2258] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:49.105813 20043 ts_tablet_manager.cc:1905] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:16:49.106153 20041 ts_tablet_manager.cc:1905] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:16:49.107638 20043 ts_tablet_manager.cc:1918] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:16:49.107708 20043 log.cc:1196] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/1c8ffc54f21b4bc99673ed5ccb769c2e
I20260927 11:16:49.107795 20041 ts_tablet_manager.cc:1918] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:16:49.107848 20041 log.cc:1196] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/1c8ffc54f21b4bc99673ed5ccb769c2e
I20260927 11:16:49.108014 20043 ts_tablet_manager.cc:1939] T 1c8ffc54f21b4bc99673ed5ccb769c2e P ca4d9b3986f6496eb76572cae18044fc: Deleting consensus metadata
I20260927 11:16:49.108119 20041 ts_tablet_manager.cc:1939] T 1c8ffc54f21b4bc99673ed5ccb769c2e P 76dd51fd5aee43b5ac424ed56eb89d0d: Deleting consensus metadata
I20260927 11:16:49.108412 19533 catalog_manager.cc:4841] TS ca4d9b3986f6496eb76572cae18044fc (127.0.98.2:32891): tablet 1c8ffc54f21b4bc99673ed5ccb769c2e (table test-table [id=404577447fd9458ab75e7ccfb4ef7dae]) successfully deleted
I20260927 11:16:49.110220 19532 catalog_manager.cc:4841] TS 76dd51fd5aee43b5ac424ed56eb89d0d (127.0.98.3:45501): tablet 1c8ffc54f21b4bc99673ed5ccb769c2e (table test-table [id=404577447fd9458ab75e7ccfb4ef7dae]) successfully deleted
W20260927 11:16:49.110378 19533 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40139 (error 108)
I20260927 11:16:49.110653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 19742
I20260927 11:16:49.110786 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.110647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40139 (local address 127.0.0.1:50014)
0927 11:16:49.110701 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.110754 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 11:16:49.112387 19533 catalog_manager.cc:4584] TS ec89a8387ff2463a831f76624d0fbe52 (127.0.98.1:40139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c8ffc54f21b4bc99673ed5ccb769c2e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40139: connect: Connection refused (error 111)
I20260927 11:16:49.115439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 19876
I20260927 11:16:49.119644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 19514
2026-09-27T11:16:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1818 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T11:16:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:16:49Z Disabled control of system clock
I20260927 11:16:49.145761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46365
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:16:49.223472 20048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:49.223663 20048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:49.223683 20048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:49.224720 20048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:49.224751 20048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:49.224763 20048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:49.224776 20048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:49.224788 20048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:49.226004 20048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46365
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:49.226263 20048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:49.226531 20048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:49.231003 20056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:49.231009 20053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:49.231007 20054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:49.231370 20048 server_base.cc:1034] running on GCE node
I20260927 11:16:49.231647 20048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:49.231926 20048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:49.233126 20048 hybrid_clock.cc:648] HybridClock initialized: now 1790507809233111 us; error 24 us; skew 500 ppm
I20260927 11:16:49.234328 20048 webserver.cc:460] Webserver started at http://127.0.98.62:45217/ using document root <none> and password file <none>
I20260927 11:16:49.234548 20048 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:49.234615 20048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:49.234768 20048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:49.235749 20048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "3edf7016c64c448caccb0d921bf915b9"
format_stamp: "Formatted at 2026-09-27 11:16:49 on dist-test-slave-9hq9"
server_key: "649b90e0b168bbed66eac4226de56cec"
server_key_iv: "3e5c8355da005d97d0f1549055b16f96"
server_key_version: "encryptionkey@0"
I20260927 11:16:49.236124 20048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "3edf7016c64c448caccb0d921bf915b9"
format_stamp: "Formatted at 2026-09-27 11:16:49 on dist-test-slave-9hq9"
server_key: "649b90e0b168bbed66eac4226de56cec"
server_key_iv: "3e5c8355da005d97d0f1549055b16f96"
server_key_version: "encryptionkey@0"
I20260927 11:16:49.237342 20048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:16:49.237987 20062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:49.238168 20048 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:49.238267 20048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "3edf7016c64c448caccb0d921bf915b9"
format_stamp: "Formatted at 2026-09-27 11:16:49 on dist-test-slave-9hq9"
server_key: "649b90e0b168bbed66eac4226de56cec"
server_key_iv: "3e5c8355da005d97d0f1549055b16f96"
server_key_version: "encryptionkey@0"
I20260927 11:16:49.238350 20048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:49.247128 20048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:49.247392 20048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:49.247529 20048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:49.251180 20048 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46365
I20260927 11:16:49.251220 20114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46365 every 8 connection(s)
I20260927 11:16:49.251510 20048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:49.252367 20115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:49.254449 20115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9: Bootstrap starting.
I20260927 11:16:49.254973 20115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:49.255262 20115 log.cc:824] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:49.255899 20115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9: No bootstrap required, opened a new log
I20260927 11:16:49.257411 20115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edf7016c64c448caccb0d921bf915b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46365 } }
I20260927 11:16:49.257530 20115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:49.257552 20115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3edf7016c64c448caccb0d921bf915b9, State: Initialized, Role: FOLLOWER
I20260927 11:16:49.257737 20115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edf7016c64c448caccb0d921bf915b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46365 } }
I20260927 11:16:49.257812 20115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:49.257831 20115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:49.257889 20115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:49.258462 20115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edf7016c64c448caccb0d921bf915b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46365 } }
I20260927 11:16:49.258566 20115 leader_election.cc:302] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3edf7016c64c448caccb0d921bf915b9; no voters:
I20260927 11:16:49.258721 20115 leader_election.cc:288] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:16:49.258773 20118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:49.258951 20118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [term 1 LEADER]: Becoming Leader. State: Replica: 3edf7016c64c448caccb0d921bf915b9, State: Running, Role: LEADER
I20260927 11:16:49.258977 20115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:49.259047 20118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edf7016c64c448caccb0d921bf915b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46365 } }
I20260927 11:16:49.259588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 20048
I20260927 11:16:49.259562 20119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3edf7016c64c448caccb0d921bf915b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edf7016c64c448caccb0d921bf915b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46365 } } }
I20260927 11:16:49.259660 20119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:49.259720 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:16:49.259821 20120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3edf7016c64c448caccb0d921bf915b9. Latest consensus state: current_term: 1 leader_uuid: "3edf7016c64c448caccb0d921bf915b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3edf7016c64c448caccb0d921bf915b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46365 } } }
I20260927 11:16:49.259879 20120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:49.259917 392 external_mini_cluster.cc:1428] Setting key 4eb1baca9b4291c74cc0ee0847cf46c6
W20260927 11:16:49.260826 20134 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:16:49.260883 20134 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:16:49.260936 20127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:49.261085 20127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:16:49.262482 20127 catalog_manager.cc:1347] Generated new cluster ID: 12c1766361d64dc5b6c7efccb79afded
I20260927 11:16:49.262534 20127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:16:49.265100 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.260543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.0.98.62:46365)
0927 11:16:49.260743 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.260750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.260783 (+ 33us) server_negotiation.cc:407] Connection header received
0927 11:16:49.260866 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.260871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.260970 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.261104 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:49.261324 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.262365 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.263230 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.263660 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.263798 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.264447 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:49.264450 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:49.264451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:49.264618 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:49.264647 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:49.264657 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:49.264728 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.264825 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.264902 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":104,"threads_started":1}
I20260927 11:16:49.278138 20127 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:16:49.278627 20127 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:49.282243 20127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9: Generated new TSK 0
I20260927 11:16:49.282404 20127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:16:49.290551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46365
--builtin_ntp_servers=127.0.98.20:37753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:16:49.367594 20139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:49.367769 20139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:49.367789 20139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:49.368891 20139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:49.368927 20139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:49.368952 20139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 11:16:49.370354 20139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:49.370565 20139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:49.370855 20139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:49.375488 20144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:49.375458 20145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:49.375458 20147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:49.375873 20139 server_base.cc:1034] running on GCE node
I20260927 11:16:49.376094 20139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:49.376325 20139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:49.377477 20139 hybrid_clock.cc:648] HybridClock initialized: now 1790507809377460 us; error 29 us; skew 500 ppm
I20260927 11:16:49.378831 20139 webserver.cc:460] Webserver started at http://127.0.98.1:35605/ using document root <none> and password file <none>
I20260927 11:16:49.379074 20139 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:49.379127 20139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:49.379247 20139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:49.380489 20139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/data/instance:
uuid: "1cf52fcbd5754f50b9e3553d76ff218c"
format_stamp: "Formatted at 2026-09-27 11:16:49 on dist-test-slave-9hq9"
server_key: "7ead8bc15f024c1b7d17de3966c6beae"
server_key_iv: "40bf941780f33ad2633c7e678fb3549a"
server_key_version: "encryptionkey@0"
I20260927 11:16:49.380853 20139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1cf52fcbd5754f50b9e3553d76ff218c"
format_stamp: "Formatted at 2026-09-27 11:16:49 on dist-test-slave-9hq9"
server_key: "7ead8bc15f024c1b7d17de3966c6beae"
server_key_iv: "40bf941780f33ad2633c7e678fb3549a"
server_key_version: "encryptionkey@0"
I20260927 11:16:49.382462 20139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 11:16:49.383267 20153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:49.383448 20139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:49.383528 20139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/wal
uuid: "1cf52fcbd5754f50b9e3553d76ff218c"
format_stamp: "Formatted at 2026-09-27 11:16:49 on dist-test-slave-9hq9"
server_key: "7ead8bc15f024c1b7d17de3966c6beae"
server_key_iv: "40bf941780f33ad2633c7e678fb3549a"
server_key_version: "encryptionkey@0"
I20260927 11:16:49.383608 20139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:49.403088 20139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:49.403301 20139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:49.403390 20139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:49.403540 20139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:49.403771 20139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:49.403800 20139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:49.403818 20139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:49.403831 20139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:49.409293 20139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33453
I20260927 11:16:49.409397 20266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33453 every 8 connection(s)
I20260927 11:16:49.409654 20139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/data/info.pb
I20260927 11:16:49.414191 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.410322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40571 (local address 127.0.98.62:46365)
0927 11:16:49.410360 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.410362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.410558 (+ 196us) server_negotiation.cc:407] Connection header received
0927 11:16:49.411008 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.411009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.411019 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.411066 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:49.411618 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.412114 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.412994 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.413176 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.413224 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.413961 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:49.413961 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:49.413962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:49.413965 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:49.413974 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:49.413978 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:49.414013 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.414097 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.414119 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:16:49.414417 20269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.410313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46365 (local address 127.0.98.1:40571)
0927 11:16:49.410468 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.410497 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:49.410928 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:49.411112 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:49.411115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:49.411122 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:49.411549 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.411553 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.412188 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.412189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.412923 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.412925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.413242 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.413242 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.413349 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.413850 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:16:49.413850 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:49.413874 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:16:49.413893 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:49.414014 (+ 121us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:49.414016 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:49.414018 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:16:49.414111 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 11:16:49.414189 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260927 11:16:49.414847 20267 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46365
I20260927 11:16:49.414913 20267 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:49.415083 20267 heartbeater.cc:511] Master 127.0.98.62:46365 requested a full tablet report, sending...
I20260927 11:16:49.415314 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 20139
I20260927 11:16:49.415407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-0/wal/instance
I20260927 11:16:49.415612 392 external_mini_cluster.cc:1428] Setting key 5487a1eb75286631573df4134cec9484
I20260927 11:16:49.415601 20079 ts_manager.cc:194] Registered new tserver with Master: 1cf52fcbd5754f50b9e3553d76ff218c (127.0.98.1:33453)
I20260927 11:16:49.416265 20079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40571
I20260927 11:16:49.416782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46365
--builtin_ntp_servers=127.0.98.20:37753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:16:49.493438 20270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:49.493633 20270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:49.493681 20270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:49.494812 20270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:49.494868 20270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:49.494906 20270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 11:16:49.496151 20270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:49.496371 20270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:49.496644 20270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:49.501237 20278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:49.501260 20275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:49.501425 20276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:49.501543 20270 server_base.cc:1034] running on GCE node
I20260927 11:16:49.501708 20270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:49.501890 20270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:49.503057 20270 hybrid_clock.cc:648] HybridClock initialized: now 1790507809503032 us; error 33 us; skew 500 ppm
I20260927 11:16:49.504179 20270 webserver.cc:460] Webserver started at http://127.0.98.2:37937/ using document root <none> and password file <none>
I20260927 11:16:49.504416 20270 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:49.504483 20270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:49.504626 20270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:49.505577 20270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/data/instance:
uuid: "f2de084c9b7e464b97b737d7bf1bcbaa"
format_stamp: "Formatted at 2026-09-27 11:16:49 on dist-test-slave-9hq9"
server_key: "3592a6bf17c979af03b76e37259f1bb5"
server_key_iv: "6d161a31fb3348f1cd6392f1e5bff0db"
server_key_version: "encryptionkey@0"
I20260927 11:16:49.505910 20270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f2de084c9b7e464b97b737d7bf1bcbaa"
format_stamp: "Formatted at 2026-09-27 11:16:49 on dist-test-slave-9hq9"
server_key: "3592a6bf17c979af03b76e37259f1bb5"
server_key_iv: "6d161a31fb3348f1cd6392f1e5bff0db"
server_key_version: "encryptionkey@0"
I20260927 11:16:49.507103 20270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:16:49.507884 20284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:49.508049 20270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:49.508157 20270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/wal
uuid: "f2de084c9b7e464b97b737d7bf1bcbaa"
format_stamp: "Formatted at 2026-09-27 11:16:49 on dist-test-slave-9hq9"
server_key: "3592a6bf17c979af03b76e37259f1bb5"
server_key_iv: "6d161a31fb3348f1cd6392f1e5bff0db"
server_key_version: "encryptionkey@0"
I20260927 11:16:49.508247 20270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:49.516836 20270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:49.517076 20270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:49.517206 20270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:49.517398 20270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:49.517678 20270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:49.517732 20270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:49.517769 20270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:49.517808 20270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:49.523484 20270 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46445
I20260927 11:16:49.523551 20397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46445 every 8 connection(s)
I20260927 11:16:49.523850 20270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/data/info.pb
I20260927 11:16:49.528798 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.524490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44445 (local address 127.0.98.62:46365)
0927 11:16:49.524555 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.524556 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.524760 (+ 204us) server_negotiation.cc:407] Connection header received
0927 11:16:49.525247 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.525249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.525259 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.525327 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:49.525952 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.526513 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.527430 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.527637 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.527719 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.528523 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:49.528525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:49.528525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:49.528530 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:49.528545 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:49.528549 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:49.528597 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.528698 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.528723 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 11:16:49.529003 20400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.524466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46365 (local address 127.0.98.2:44445)
0927 11:16:49.524656 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.524685 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:49.525146 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:49.525393 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:49.525396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:49.525412 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:49.525853 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.525858 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.526602 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.526603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.527343 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.527345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.527730 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.527730 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.527860 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.528386 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:16:49.528386 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:49.528412 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:16:49.528433 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:49.528596 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:49.528602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:49.528605 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:49.528686 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 11:16:49.528777 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260927 11:16:49.529537 20398 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46365
I20260927 11:16:49.529604 20398 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:49.529769 20398 heartbeater.cc:511] Master 127.0.98.62:46365 requested a full tablet report, sending...
I20260927 11:16:49.530138 20079 ts_manager.cc:194] Registered new tserver with Master: f2de084c9b7e464b97b737d7bf1bcbaa (127.0.98.2:46445)
I20260927 11:16:49.530550 20079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44445
I20260927 11:16:49.531194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 20270
I20260927 11:16:49.531301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/wal/instance
I20260927 11:16:49.531499 392 external_mini_cluster.cc:1428] Setting key 1fb88c953de35385299d441d0fb5319f
I20260927 11:16:49.532583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46365
--builtin_ntp_servers=127.0.98.20:37753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 11:16:49.610324 20401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:49.610541 20401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:49.610564 20401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:49.611660 20401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:49.611696 20401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:49.611719 20401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 11:16:49.612994 20401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:49.613163 20401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:49.613389 20401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:49.617897 20406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:49.617954 20401 server_base.cc:1034] running on GCE node
W20260927 11:16:49.617908 20407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:49.617928 20409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:49.618299 20401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:49.618562 20401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:49.619707 20401 hybrid_clock.cc:648] HybridClock initialized: now 1790507809619694 us; error 25 us; skew 500 ppm
I20260927 11:16:49.620882 20401 webserver.cc:460] Webserver started at http://127.0.98.3:40475/ using document root <none> and password file <none>
I20260927 11:16:49.621115 20401 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:49.621181 20401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:49.621317 20401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:49.622360 20401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/data/instance:
uuid: "4313e61dc46a460292d2e6e139416ca7"
format_stamp: "Formatted at 2026-09-27 11:16:49 on dist-test-slave-9hq9"
server_key: "8b85bb8809ef610148637982a68da04a"
server_key_iv: "8ad17f202652724db8e8e216b18398d0"
server_key_version: "encryptionkey@0"
I20260927 11:16:49.622725 20401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4313e61dc46a460292d2e6e139416ca7"
format_stamp: "Formatted at 2026-09-27 11:16:49 on dist-test-slave-9hq9"
server_key: "8b85bb8809ef610148637982a68da04a"
server_key_iv: "8ad17f202652724db8e8e216b18398d0"
server_key_version: "encryptionkey@0"
I20260927 11:16:49.623960 20401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:16:49.624673 20415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:49.624835 20401 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:49.624926 20401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/wal
uuid: "4313e61dc46a460292d2e6e139416ca7"
format_stamp: "Formatted at 2026-09-27 11:16:49 on dist-test-slave-9hq9"
server_key: "8b85bb8809ef610148637982a68da04a"
server_key_iv: "8ad17f202652724db8e8e216b18398d0"
server_key_version: "encryptionkey@0"
I20260927 11:16:49.625010 20401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:49.637416 20401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:49.637671 20401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:49.637820 20401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:49.638019 20401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 11:16:49.638332 20401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 11:16:49.638386 20401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:49.638424 20401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 11:16:49.638451 20401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:49.644290 20401 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41527
I20260927 11:16:49.644346 20528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41527 every 8 connection(s)
I20260927 11:16:49.644604 20401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/data/info.pb
I20260927 11:16:49.647209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 20401
I20260927 11:16:49.647320 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/wal/instance
I20260927 11:16:49.647521 392 external_mini_cluster.cc:1428] Setting key a1af91a223c54b2b624953a88ca78a60
I20260927 11:16:49.649518 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.645194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50655 (local address 127.0.98.62:46365)
0927 11:16:49.645254 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.645256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.645509 (+ 253us) server_negotiation.cc:407] Connection header received
0927 11:16:49.645939 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.645940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.645950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.646001 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:49.646619 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.647103 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.648270 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.648550 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.648602 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.649341 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:49.649342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:49.649342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:49.649345 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:49.649355 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:49.649358 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:49.649395 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.649443 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.649463 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 11:16:49.649744 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.645189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46365 (local address 127.0.98.3:50655)
0927 11:16:49.645343 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.645392 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:49.645853 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:49.646074 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:49.646077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:49.646084 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:16:49.646544 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.646548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.647200 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.647201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.648177 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.648179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.648623 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.648624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.648728 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.649231 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 11:16:49.649231 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:16:49.649256 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 11:16:49.649275 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:16:49.649397 (+ 122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:16:49.649399 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:16:49.649401 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:16:49.649445 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 11:16:49.649511 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":83,"threads_started":1}
I20260927 11:16:49.650295 20529 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46365
I20260927 11:16:49.650368 20529 heartbeater.cc:464] Registering TS with master...
I20260927 11:16:49.650489 20529 heartbeater.cc:511] Master 127.0.98.62:46365 requested a full tablet report, sending...
I20260927 11:16:49.650800 20079 ts_manager.cc:194] Registered new tserver with Master: 4313e61dc46a460292d2e6e139416ca7 (127.0.98.3:41527)
I20260927 11:16:49.651116 20079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50655
I20260927 11:16:49.659333 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 11:16:49.663324 20136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.660747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39494 (local address 127.0.98.62:46365)
0927 11:16:49.660809 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.660810 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.660903 (+ 93us) server_negotiation.cc:407] Connection header received
0927 11:16:49.660967 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.660967 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.660977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.661027 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:49.661147 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.661588 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.662333 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.662498 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.662549 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.663157 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:49.663158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:49.663158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:49.663162 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:49.663171 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:49.663175 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:49.663211 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.663259 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.663277 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:16:49.664595 20079 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 11:16:49.666131 20079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 11:16:49.672120 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.668106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41527 (local address 127.0.0.1:57402)
0927 11:16:49.668399 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.668420 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:49.668523 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:49.668790 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:49.668793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:49.668795 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:49.668909 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.668911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.670250 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.670251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.671173 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.671176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.671851 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.671851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.671926 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.671938 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 11:16:49.672010 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 11:16:49.672057 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":26,"thread_start_us":75,"threads_started":1}
I20260927 11:16:49.672130 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.668168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57402 (local address 127.0.98.3:41527)
0927 11:16:49.668333 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.668336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.668493 (+ 157us) server_negotiation.cc:407] Connection header received
0927 11:16:49.668613 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.668616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.668631 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.668684 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:49.668996 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.670143 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.671239 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.671766 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.671849 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.671937 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.672023 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.672055 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260927 11:16:49.672276 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.668297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33453 (local address 127.0.0.1:36518)
0927 11:16:49.668507 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.668516 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:49.668557 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:49.668839 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:49.668840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:49.668842 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:49.668889 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.668893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.669859 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.669860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.671107 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.671109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.672010 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.672011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.672074 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.672078 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 11:16:49.672110 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 11:16:49.672150 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260927 11:16:49.672626 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.668517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.0.98.1:33453)
0927 11:16:49.668683 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.668688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.668696 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:16:49.668732 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.668734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.668747 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.668989 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:49.669050 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.669764 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.671192 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.671907 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.672395 (+ 488us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.672487 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.672511 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.672549 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260927 11:16:49.672819 20463 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2784f340374183a9beccb61277d509 (DEFAULT_TABLE table=test-table [id=39e29d5bbd9a49558d8b2a7e202cdc14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:49.673153 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.668076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46445 (local address 127.0.0.1:45278)
0927 11:16:49.668378 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.668418 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:49.668480 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:49.668790 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:49.668799 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:49.668800 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:49.668850 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.668852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.670240 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.670241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.671317 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.671319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.672576 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.672577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.672643 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.672646 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:49.673050 (+ 404us) client_negotiation.cc:240] Negotiation successful
0927 11:16:49.673091 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":31,"thread_start_us":45,"threads_started":1}
I20260927 11:16:49.673182 20463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2784f340374183a9beccb61277d509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:49.673310 20201 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2784f340374183a9beccb61277d509 (DEFAULT_TABLE table=test-table [id=39e29d5bbd9a49558d8b2a7e202cdc14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:49.673552 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.668193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45278 (local address 127.0.98.2:46445)
0927 11:16:49.668626 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.668631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.668640 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:16:49.668678 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.668681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.668696 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.668777 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:49.669005 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.670144 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.671414 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.672086 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.672142 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.672949 (+ 807us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.673371 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.673416 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":90,"threads_started":1}
I20260927 11:16:49.673652 20201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2784f340374183a9beccb61277d509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:49.674417 20332 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2784f340374183a9beccb61277d509 (DEFAULT_TABLE table=test-table [id=39e29d5bbd9a49558d8b2a7e202cdc14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 11:16:49.674723 20332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2784f340374183a9beccb61277d509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:49.676249 20544 tablet_bootstrap.cc:492] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c: Bootstrap starting.
I20260927 11:16:49.676379 20543 tablet_bootstrap.cc:492] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7: Bootstrap starting.
I20260927 11:16:49.676837 20544 tablet_bootstrap.cc:654] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:49.676990 20543 tablet_bootstrap.cc:654] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:49.677150 20544 log.cc:824] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:49.677275 20543 log.cc:824] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:49.677925 20543 tablet_bootstrap.cc:492] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7: No bootstrap required, opened a new log
I20260927 11:16:49.677933 20544 tablet_bootstrap.cc:492] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c: No bootstrap required, opened a new log
I20260927 11:16:49.677968 20547 tablet_bootstrap.cc:492] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa: Bootstrap starting.
I20260927 11:16:49.677995 20544 ts_tablet_manager.cc:1397] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:16:49.678115 20543 ts_tablet_manager.cc:1397] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 11:16:49.678627 20547 tablet_bootstrap.cc:654] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:49.678969 20547 log.cc:824] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:49.679695 20547 tablet_bootstrap.cc:492] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa: No bootstrap required, opened a new log
I20260927 11:16:49.679762 20547 ts_tablet_manager.cc:1397] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 11:16:49.680395 20543 raft_consensus.cc:348] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2de084c9b7e464b97b737d7bf1bcbaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46445 } } peers { permanent_uuid: "4313e61dc46a460292d2e6e139416ca7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "1cf52fcbd5754f50b9e3553d76ff218c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260927 11:16:49.680536 20543 raft_consensus.cc:374] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:49.680420 20544 raft_consensus.cc:348] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2de084c9b7e464b97b737d7bf1bcbaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46445 } } peers { permanent_uuid: "4313e61dc46a460292d2e6e139416ca7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "1cf52fcbd5754f50b9e3553d76ff218c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260927 11:16:49.680576 20543 raft_consensus.cc:729] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4313e61dc46a460292d2e6e139416ca7, State: Initialized, Role: FOLLOWER
I20260927 11:16:49.680586 20544 raft_consensus.cc:374] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:49.680624 20544 raft_consensus.cc:729] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf52fcbd5754f50b9e3553d76ff218c, State: Initialized, Role: FOLLOWER
I20260927 11:16:49.680713 20543 consensus_queue.cc:260] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2de084c9b7e464b97b737d7bf1bcbaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46445 } } peers { permanent_uuid: "4313e61dc46a460292d2e6e139416ca7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "1cf52fcbd5754f50b9e3553d76ff218c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260927 11:16:49.680730 20544 consensus_queue.cc:260] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2de084c9b7e464b97b737d7bf1bcbaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46445 } } peers { permanent_uuid: "4313e61dc46a460292d2e6e139416ca7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "1cf52fcbd5754f50b9e3553d76ff218c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260927 11:16:49.680963 20544 ts_tablet_manager.cc:1428] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 11:16:49.680982 20529 heartbeater.cc:503] Master 127.0.98.62:46365 was elected leader, sending a full tablet report...
I20260927 11:16:49.681084 20267 heartbeater.cc:503] Master 127.0.98.62:46365 was elected leader, sending a full tablet report...
I20260927 11:16:49.680970 20543 ts_tablet_manager.cc:1428] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:16:49.682217 20547 raft_consensus.cc:348] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2de084c9b7e464b97b737d7bf1bcbaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46445 } } peers { permanent_uuid: "4313e61dc46a460292d2e6e139416ca7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "1cf52fcbd5754f50b9e3553d76ff218c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260927 11:16:49.682360 20547 raft_consensus.cc:374] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:49.682410 20547 raft_consensus.cc:729] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2de084c9b7e464b97b737d7bf1bcbaa, State: Initialized, Role: FOLLOWER
I20260927 11:16:49.682524 20547 consensus_queue.cc:260] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2de084c9b7e464b97b737d7bf1bcbaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46445 } } peers { permanent_uuid: "4313e61dc46a460292d2e6e139416ca7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "1cf52fcbd5754f50b9e3553d76ff218c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260927 11:16:49.682756 20547 ts_tablet_manager.cc:1428] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 11:16:49.682833 20398 heartbeater.cc:503] Master 127.0.98.62:46365 was elected leader, sending a full tablet report...
I20260927 11:16:49.690757 20551 raft_consensus.cc:482] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 11:16:49.690821 20551 raft_consensus.cc:504] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2de084c9b7e464b97b737d7bf1bcbaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46445 } } peers { permanent_uuid: "4313e61dc46a460292d2e6e139416ca7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "1cf52fcbd5754f50b9e3553d76ff218c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260927 11:16:49.691025 20551 leader_election.cc:288] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4313e61dc46a460292d2e6e139416ca7 (127.0.98.3:41527), 1cf52fcbd5754f50b9e3553d76ff218c (127.0.98.1:33453)
I20260927 11:16:49.693624 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.691232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50671 (local address 127.0.98.3:41527)
0927 11:16:49.691271 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.691273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.691282 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:16:49.691355 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.691356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.691364 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.691421 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:49.691575 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.692036 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.692941 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.693268 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.693363 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.693371 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.693524 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.693554 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 11:16:49.693629 20400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.691152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41527 (local address 127.0.98.2:50671)
0927 11:16:49.691209 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.691219 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:49.691282 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:49.691434 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:49.691435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:49.691437 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:49.691489 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.691491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.692106 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.692106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.692842 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.692844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.693366 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.693367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.693434 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.693437 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 11:16:49.693517 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 11:16:49.693554 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 11:16:49.693905 20552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.691260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33453 (local address 127.0.98.2:50875)
0927 11:16:49.691412 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.691420 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 11:16:49.691486 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:16:49.691596 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:16:49.691596 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:16:49.691598 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 11:16:49.691654 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.691656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.692303 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.692304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.693022 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:16:49.693024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.693536 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.693537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:16:49.693633 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.693635 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 11:16:49.693698 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 11:16:49.693739 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260927 11:16:49.693982 20483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2784f340374183a9beccb61277d509" candidate_uuid: "f2de084c9b7e464b97b737d7bf1bcbaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4313e61dc46a460292d2e6e139416ca7" is_pre_election: true
I20260927 11:16:49.694079 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.691341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50875 (local address 127.0.98.1:33453)
0927 11:16:49.691393 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.691395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.691478 (+ 83us) server_negotiation.cc:407] Connection header received
0927 11:16:49.691535 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.691536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.691544 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.691593 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 11:16:49.691744 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.692204 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.693106 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.693453 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.693533 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.693540 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.693976 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.694017 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:16:49.694113 20483 raft_consensus.cc:2454] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2de084c9b7e464b97b737d7bf1bcbaa in term 0.
I20260927 11:16:49.694319 20287 leader_election.cc:302] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4313e61dc46a460292d2e6e139416ca7, f2de084c9b7e464b97b737d7bf1bcbaa; no voters:
I20260927 11:16:49.694350 20221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2784f340374183a9beccb61277d509" candidate_uuid: "f2de084c9b7e464b97b737d7bf1bcbaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf52fcbd5754f50b9e3553d76ff218c" is_pre_election: true
I20260927 11:16:49.694451 20221 raft_consensus.cc:2454] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2de084c9b7e464b97b737d7bf1bcbaa in term 0.
I20260927 11:16:49.694461 20551 raft_consensus.cc:2781] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 11:16:49.694504 20551 raft_consensus.cc:482] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 11:16:49.694533 20551 raft_consensus.cc:3037] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:49.695214 20551 raft_consensus.cc:504] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2de084c9b7e464b97b737d7bf1bcbaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46445 } } peers { permanent_uuid: "4313e61dc46a460292d2e6e139416ca7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "1cf52fcbd5754f50b9e3553d76ff218c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260927 11:16:49.695315 20551 leader_election.cc:288] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [CANDIDATE]: Term 1 election: Requested vote from peers 4313e61dc46a460292d2e6e139416ca7 (127.0.98.3:41527), 1cf52fcbd5754f50b9e3553d76ff218c (127.0.98.1:33453)
I20260927 11:16:49.695451 20483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2784f340374183a9beccb61277d509" candidate_uuid: "f2de084c9b7e464b97b737d7bf1bcbaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4313e61dc46a460292d2e6e139416ca7"
I20260927 11:16:49.695487 20221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2784f340374183a9beccb61277d509" candidate_uuid: "f2de084c9b7e464b97b737d7bf1bcbaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf52fcbd5754f50b9e3553d76ff218c"
I20260927 11:16:49.695524 20483 raft_consensus.cc:3037] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:49.695544 20221 raft_consensus.cc:3037] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:49.696296 20483 raft_consensus.cc:2454] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2de084c9b7e464b97b737d7bf1bcbaa in term 1.
I20260927 11:16:49.696296 20221 raft_consensus.cc:2454] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2de084c9b7e464b97b737d7bf1bcbaa in term 1.
I20260927 11:16:49.696465 20287 leader_election.cc:302] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4313e61dc46a460292d2e6e139416ca7, f2de084c9b7e464b97b737d7bf1bcbaa; no voters:
I20260927 11:16:49.696561 20551 raft_consensus.cc:2781] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:49.696683 20551 raft_consensus.cc:686] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [term 1 LEADER]: Becoming Leader. State: Replica: f2de084c9b7e464b97b737d7bf1bcbaa, State: Running, Role: LEADER
I20260927 11:16:49.696744 20551 consensus_queue.cc:237] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2de084c9b7e464b97b737d7bf1bcbaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46445 } } peers { permanent_uuid: "4313e61dc46a460292d2e6e139416ca7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } } peers { permanent_uuid: "1cf52fcbd5754f50b9e3553d76ff218c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } }
I20260927 11:16:49.697422 20078 catalog_manager.cc:5495] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa reported cstate change: term changed from 0 to 1, leader changed from <none> to f2de084c9b7e464b97b737d7bf1bcbaa (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f2de084c9b7e464b97b737d7bf1bcbaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2de084c9b7e464b97b737d7bf1bcbaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4313e61dc46a460292d2e6e139416ca7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cf52fcbd5754f50b9e3553d76ff218c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 } health_report { overall_health: UNKNOWN } } }
I20260927 11:16:49.707010 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.704580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45280 (local address 127.0.98.2:46445)
0927 11:16:49.704646 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:49.704648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:49.704674 (+ 26us) server_negotiation.cc:407] Connection header received
0927 11:16:49.704706 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:49.704707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:49.704722 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:49.704794 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:16:49.704936 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.705477 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.706267 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:49.706459 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:49.706546 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:49.706701 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:16:49.706786 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:16:49.706878 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:49.706896 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 11:16:49.706911 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 11:16:49.709571 20221 raft_consensus.cc:1261] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c [term 1 FOLLOWER]: Refusing update from remote peer f2de084c9b7e464b97b737d7bf1bcbaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:16:49.709904 20483 raft_consensus.cc:1261] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7 [term 1 FOLLOWER]: Refusing update from remote peer f2de084c9b7e464b97b737d7bf1bcbaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:16:49.710160 20551 consensus_queue.cc:1035] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cf52fcbd5754f50b9e3553d76ff218c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:16:49.710318 20551 consensus_queue.cc:1035] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4313e61dc46a460292d2e6e139416ca7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:16:49.713477 20557 mvcc.cc:205] Tried to move back new op lower bound from 7333919988564578304 to 7333919988518043648. Current Snapshot: MvccSnapshot[applied={T|T < 7333919988564578304}]
I20260927 11:16:49.713495 20559 mvcc.cc:205] Tried to move back new op lower bound from 7333919988564578304 to 7333919988518043648. Current Snapshot: MvccSnapshot[applied={T|T < 7333919988564578304}]
I20260927 11:16:49.714926 20078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:16:49.715001 20078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 11:16:49.715615 20078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3edf7016c64c448caccb0d921bf915b9: Sending DeleteTablet for 3 replicas of tablet 4c2784f340374183a9beccb61277d509
I20260927 11:16:49.715910 20332 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2784f340374183a9beccb61277d509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:49 UTC) from {username='slave'} at 127.0.0.1:45278
I20260927 11:16:49.715961 20201 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2784f340374183a9beccb61277d509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:49 UTC) from {username='slave'} at 127.0.0.1:36518
I20260927 11:16:49.716128 20565 tablet_replica.cc:330] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa: stopping tablet replica
I20260927 11:16:49.716158 20566 tablet_replica.cc:330] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c: stopping tablet replica
I20260927 11:16:49.716244 20566 raft_consensus.cc:2229] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:16:49.716259 20565 raft_consensus.cc:2229] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [term 1 LEADER]: Raft consensus shutting down.
I20260927 11:16:49.716267 20463 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2784f340374183a9beccb61277d509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 11:16:49 UTC) from {username='slave'} at 127.0.0.1:57402
I20260927 11:16:49.716346 20566 raft_consensus.cc:2258] T 4c2784f340374183a9beccb61277d509 P 1cf52fcbd5754f50b9e3553d76ff218c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:49.716405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 20139
I20260927 11:16:49.716516 20567 tablet_replica.cc:330] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7: stopping tablet replica
I20260927 11:16:49.716595 20567 raft_consensus.cc:2229] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 11:16:49.716678 20567 raft_consensus.cc:2258] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:49.717060 20567 ts_tablet_manager.cc:1905] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:16:49.718195 20565 raft_consensus.cc:2258] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 11:16:49.718824 20565 ts_tablet_manager.cc:1905] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 11:16:49.720656 20565 ts_tablet_manager.cc:1918] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:16:49.720745 20565 log.cc:1196] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-1/wal/wals/4c2784f340374183a9beccb61277d509
I20260927 11:16:49.720983 20565 ts_tablet_manager.cc:1939] T 4c2784f340374183a9beccb61277d509 P f2de084c9b7e464b97b737d7bf1bcbaa: Deleting consensus metadata
I20260927 11:16:49.721339 20063 catalog_manager.cc:4841] TS f2de084c9b7e464b97b737d7bf1bcbaa (127.0.98.2:46445): tablet 4c2784f340374183a9beccb61277d509 (table test-table [id=39e29d5bbd9a49558d8b2a7e202cdc14]) successfully deleted
I20260927 11:16:49.721383 20567 ts_tablet_manager.cc:1918] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 11:16:49.721437 20567 log.cc:1196] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7: Deleting WAL directory at /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790507576023819-392-0/minicluster-data/ts-2/wal/wals/4c2784f340374183a9beccb61277d509
I20260927 11:16:49.721689 20567 ts_tablet_manager.cc:1939] T 4c2784f340374183a9beccb61277d509 P 4313e61dc46a460292d2e6e139416ca7: Deleting consensus metadata
I20260927 11:16:49.722087 20065 catalog_manager.cc:4841] TS 4313e61dc46a460292d2e6e139416ca7 (127.0.98.3:41527): tablet 4c2784f340374183a9beccb61277d509 (table test-table [id=39e29d5bbd9a49558d8b2a7e202cdc14]) successfully deleted
W20260927 11:16:49.723184 20065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33453 (error 108)
I20260927 11:16:49.723417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 20270
I20260927 11:16:49.724035 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:49.723556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33453 (local address 127.0.0.1:36530)
0927 11:16:49.723959 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 11:16:49.723999 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384}
W20260927 11:16:49.724220 20065 catalog_manager.cc:4584] TS 1cf52fcbd5754f50b9e3553d76ff218c (127.0.98.1:33453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c2784f340374183a9beccb61277d509: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33453: connect: Connection refused (error 111)
I20260927 11:16:49.727864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 20401
I20260927 11:16:49.732232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 20048
2026-09-27T11:16:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (612 ms)
[----------] 6 tests from EncryptionPolicyTest (7911 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:16:49 dist-test-slave-9hq9 krb5kdc[20571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:16:49 dist-test-slave-9hq9 krb5kdc[20571](info): set up 2 sockets
Sep 27 11:16:49 dist-test-slave-9hq9 krb5kdc[20571](info): commencing operation
krb5kdc: starting...
W20260927 11:16:50.745919 392 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:16:50 dist-test-slave-9hq9 krb5kdc[20571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:16:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:16:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:16:50.853968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:50.932114 20584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:50.932331 20584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:50.932366 20584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:50.933506 20584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:50.933573 20584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:50.933596 20584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:50.933626 20584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:50.933656 20584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:50.935072 20584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:50.935355 20584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:50.935576 20584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:50.939926 20589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:50.939941 20590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:50.940196 20584 server_base.cc:1034] running on GCE node
W20260927 11:16:50.939941 20592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:50.940454 20584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:50.940783 20584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:50.941938 20584 hybrid_clock.cc:648] HybridClock initialized: now 1790507810941918 us; error 37 us; skew 500 ppm
Sep 27 11:16:50 dist-test-slave-9hq9 krb5kdc[20571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:50.944274 20584 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:16:50.944708 20584 webserver.cc:460] Webserver started at http://127.0.0.1:44855/ using document root <none> and password file <none>
I20260927 11:16:50.944902 20584 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:50.944963 20584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:50.945101 20584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:50.946125 20584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "0f959e715cad444aa0605ac5236fe4f5"
format_stamp: "Formatted at 2026-09-27 11:16:50 on dist-test-slave-9hq9"
server_key: "da63171da2b4f47d5863117949f0484c"
server_key_iv: "d4bd4bbc25404f8759ef138d235ea68e"
server_key_version: "encryptionkey@0"
I20260927 11:16:50.946481 20584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "0f959e715cad444aa0605ac5236fe4f5"
format_stamp: "Formatted at 2026-09-27 11:16:50 on dist-test-slave-9hq9"
server_key: "da63171da2b4f47d5863117949f0484c"
server_key_iv: "d4bd4bbc25404f8759ef138d235ea68e"
server_key_version: "encryptionkey@0"
I20260927 11:16:50.947741 20584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:16:50.948416 20599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:50.948594 20584 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:16:50.948684 20584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "0f959e715cad444aa0605ac5236fe4f5"
format_stamp: "Formatted at 2026-09-27 11:16:50 on dist-test-slave-9hq9"
server_key: "da63171da2b4f47d5863117949f0484c"
server_key_iv: "d4bd4bbc25404f8759ef138d235ea68e"
server_key_version: "encryptionkey@0"
I20260927 11:16:50.948760 20584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:50.958535 20584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:50.960623 20584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:50.960763 20584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:50.964429 20584 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40225
I20260927 11:16:50.964459 20651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40225 every 8 connection(s)
I20260927 11:16:50.964756 20584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:50.965520 20652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:50.967604 20652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5: Bootstrap starting.
I20260927 11:16:50.967928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 20584
I20260927 11:16:50.968057 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:16:50.968206 20652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:50.968230 392 external_mini_cluster.cc:1428] Setting key f0493d37889ede5772493b5363da6266
I20260927 11:16:50.968526 20652 log.cc:824] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:50.969200 20652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5: No bootstrap required, opened a new log
I20260927 11:16:50.971048 20652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f959e715cad444aa0605ac5236fe4f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40225 } }
I20260927 11:16:50.971195 20652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:50.971241 20652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f959e715cad444aa0605ac5236fe4f5, State: Initialized, Role: FOLLOWER
I20260927 11:16:50.971385 20652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f959e715cad444aa0605ac5236fe4f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40225 } }
I20260927 11:16:50.971480 20652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:50.971524 20652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:50.971570 20652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:50.972055 20652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f959e715cad444aa0605ac5236fe4f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40225 } }
I20260927 11:16:50.972167 20652 leader_election.cc:302] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f959e715cad444aa0605ac5236fe4f5; no voters:
I20260927 11:16:50.972353 20652 leader_election.cc:288] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:16:50.972381 20657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:50.972488 20657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [term 1 LEADER]: Becoming Leader. State: Replica: 0f959e715cad444aa0605ac5236fe4f5, State: Running, Role: LEADER
I20260927 11:16:50.972599 20652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:50.972622 20657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f959e715cad444aa0605ac5236fe4f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40225 } }
I20260927 11:16:50.972934 20657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f959e715cad444aa0605ac5236fe4f5. Latest consensus state: current_term: 1 leader_uuid: "0f959e715cad444aa0605ac5236fe4f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f959e715cad444aa0605ac5236fe4f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40225 } } }
I20260927 11:16:50.973009 20657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:50.973197 20663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:50.973218 20658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f959e715cad444aa0605ac5236fe4f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f959e715cad444aa0605ac5236fe4f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40225 } } }
I20260927 11:16:50.973388 20658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:50.973668 20663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 11:16:50 dist-test-slave-9hq9 krb5kdc[20571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 11:16:50.975345 20663 catalog_manager.cc:1347] Generated new cluster ID: 862295ecabc048bda7367c08d07a9596
I20260927 11:16:50.975396 20663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:16:50.978538 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:50.968712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34133 (local address 127.0.0.1:40225)
0927 11:16:50.968881 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:50.968890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:50.968928 (+ 38us) server_negotiation.cc:407] Connection header received
0927 11:16:50.969380 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:50.969386 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:50.969428 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:50.969569 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:50.970239 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:50.971084 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:50.971967 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:50.972518 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:50.972920 (+ 402us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:50.975369 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:50.975372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:50.975372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:50.975382 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:50.977015 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:50.977395 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:50.977398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:50.977399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:50.977487 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:50.977720 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:50.977721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:50.977722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:50.977941 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:50.978154 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:50.978237 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 11:16:50.978331 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260927 11:16:50.979324 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:16:50.986573 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:50.980599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59459 (local address 127.0.0.1:40225)
0927 11:16:50.980631 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:50.980633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:50.980752 (+ 119us) server_negotiation.cc:407] Connection header received
0927 11:16:50.980821 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:50.980822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:50.980831 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:50.980874 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:50.981351 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:50.981808 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:50.982590 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:50.982826 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:50.982911 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:50.984032 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:50.984033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:50.984033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:50.984040 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:50.985389 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:50.985748 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:50.985749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:50.985749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:50.985800 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:50.986034 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:50.986035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:50.986035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:50.986171 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:50.986294 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:50.986413 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 11:16:50.986474 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:16:51.007236 20663 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:16:51.007745 20663 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:51.016670 20663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f959e715cad444aa0605ac5236fe4f5: Generated new TSK 0
I20260927 11:16:51.016842 20663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:16:51.020471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 20584
W20260927 11:16:51.024405 20569 connection.cc:535] client connection to 127.0.0.1:40225 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40225): Connection reset by peer (error 104)
2026-09-27T11:16:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:16:51 dist-test-slave-9hq9 krb5kdc[20683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:16:51 dist-test-slave-9hq9 krb5kdc[20683](info): set up 2 sockets
Sep 27 11:16:51 dist-test-slave-9hq9 krb5kdc[20683](info): commencing operation
krb5kdc: starting...
W20260927 11:16:52.029917 392 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:16:52 dist-test-slave-9hq9 krb5kdc[20683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:16:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:16:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:16:52.140091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45811
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:52.216367 20696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:52.216588 20696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:52.216612 20696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:52.217684 20696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:52.217716 20696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:52.217730 20696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:52.217743 20696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:52.217756 20696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:52.219130 20696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45811
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:52.219308 20696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:52.219458 20696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:52.224102 20701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:52.224139 20702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:52.224171 20696 server_base.cc:1034] running on GCE node
W20260927 11:16:52.224133 20704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:52.224453 20696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:52.224758 20696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:52.225914 20696 hybrid_clock.cc:648] HybridClock initialized: now 1790507812225898 us; error 33 us; skew 500 ppm
Sep 27 11:16:52 dist-test-slave-9hq9 krb5kdc[20683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:52.232429 20696 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:16:52.232836 20696 webserver.cc:460] Webserver started at http://127.0.0.1:33745/ using document root <none> and password file <none>
I20260927 11:16:52.233057 20696 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:52.233101 20696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:52.233266 20696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:52.234347 20696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "0aadc74ec9ed484b8578e06315d9a534"
format_stamp: "Formatted at 2026-09-27 11:16:52 on dist-test-slave-9hq9"
server_key: "93cccf64f3786c22c0a8e025d143b942"
server_key_iv: "4bbb0c49f2db932431f8884fd3746687"
server_key_version: "encryptionkey@0"
I20260927 11:16:52.234688 20696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "0aadc74ec9ed484b8578e06315d9a534"
format_stamp: "Formatted at 2026-09-27 11:16:52 on dist-test-slave-9hq9"
server_key: "93cccf64f3786c22c0a8e025d143b942"
server_key_iv: "4bbb0c49f2db932431f8884fd3746687"
server_key_version: "encryptionkey@0"
I20260927 11:16:52.235918 20696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:16:52.236593 20711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:52.236769 20696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:52.236864 20696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "0aadc74ec9ed484b8578e06315d9a534"
format_stamp: "Formatted at 2026-09-27 11:16:52 on dist-test-slave-9hq9"
server_key: "93cccf64f3786c22c0a8e025d143b942"
server_key_iv: "4bbb0c49f2db932431f8884fd3746687"
server_key_version: "encryptionkey@0"
I20260927 11:16:52.236944 20696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:52.245985 20696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:52.248144 20696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:52.248293 20696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:52.252027 20696 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45811
I20260927 11:16:52.252082 20763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45811 every 8 connection(s)
I20260927 11:16:52.252408 20696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:52.253211 20764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:52.254349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 20696
I20260927 11:16:52.254477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:16:52.254698 392 external_mini_cluster.cc:1428] Setting key b9e6e54ed9524608ea82ca0ffb699368
I20260927 11:16:52.255914 20764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534: Bootstrap starting.
I20260927 11:16:52.256529 20764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:52.256803 20764 log.cc:824] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:52.257436 20764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534: No bootstrap required, opened a new log
I20260927 11:16:52.259197 20764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aadc74ec9ed484b8578e06315d9a534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45811 } }
I20260927 11:16:52.259344 20764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:52.259393 20764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aadc74ec9ed484b8578e06315d9a534, State: Initialized, Role: FOLLOWER
I20260927 11:16:52.259527 20764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aadc74ec9ed484b8578e06315d9a534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45811 } }
I20260927 11:16:52.259624 20764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:52.259658 20764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:52.259702 20764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:52.259872 20767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:52.255160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39715 (local address 127.0.0.1:45811)
0927 11:16:52.255329 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:52.255336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:52.255359 (+ 23us) server_negotiation.cc:407] Connection header received
0927 11:16:52.255748 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:52.255753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:52.255794 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:52.255903 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:52.256138 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:52.257085 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:52.257937 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:52.258385 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:52.258512 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:52.259157 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:52.259161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:52.259161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:52.259314 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:52.259340 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:52.259350 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:52.259481 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:52.259599 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 11:16:52.259639 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260927 11:16:52.260264 20764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aadc74ec9ed484b8578e06315d9a534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45811 } }
I20260927 11:16:52.260346 20764 leader_election.cc:302] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aadc74ec9ed484b8578e06315d9a534; no voters:
I20260927 11:16:52.260527 20764 leader_election.cc:288] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:16:52.260560 20769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:52.260768 20769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [term 1 LEADER]: Becoming Leader. State: Replica: 0aadc74ec9ed484b8578e06315d9a534, State: Running, Role: LEADER
I20260927 11:16:52.260811 20764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:52.260896 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:16:52.260919 20769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aadc74ec9ed484b8578e06315d9a534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45811 } }
I20260927 11:16:52.261205 20770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aadc74ec9ed484b8578e06315d9a534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aadc74ec9ed484b8578e06315d9a534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45811 } } }
I20260927 11:16:52.261188 20769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aadc74ec9ed484b8578e06315d9a534. Latest consensus state: current_term: 1 leader_uuid: "0aadc74ec9ed484b8578e06315d9a534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aadc74ec9ed484b8578e06315d9a534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45811 } } }
I20260927 11:16:52.261260 20770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:52.261269 20769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534 [sys.catalog]: This master's current role is: LEADER
W20260927 11:16:52.262682 20788 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:16:52.262740 20788 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:16:52.262810 20776 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:52.262984 20776 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:16:52.264530 20776 catalog_manager.cc:1347] Generated new cluster ID: 946de34bc1f74ca8bdf5664a4f231dcc
I20260927 11:16:52.264581 20776 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:16:52.265645 20767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:52.262541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60573 (local address 127.0.0.1:45811)
0927 11:16:52.262581 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:52.262583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:52.262592 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:16:52.262639 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:52.262640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:52.262649 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:52.262705 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:52.262839 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:52.263509 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:52.264453 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:52.264731 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:52.264804 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:52.265435 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:52.265436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:52.265436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:52.265440 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:52.265452 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:52.265455 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:52.265507 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:52.265566 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 11:16:52.265574 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 11:16:52.285184 20776 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:16:52.285790 20776 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:52.290772 20776 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aadc74ec9ed484b8578e06315d9a534: Generated new TSK 0
I20260927 11:16:52.290942 20776 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:16:52.293848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 20696
W20260927 11:16:52.297880 20681 connection.cc:535] client connection to 127.0.0.1:45811 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45811): Connection reset by peer (error 104)
2026-09-27T11:16:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1273 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:16:52 dist-test-slave-9hq9 krb5kdc[20795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:16:52 dist-test-slave-9hq9 krb5kdc[20795](info): set up 2 sockets
Sep 27 11:16:52 dist-test-slave-9hq9 krb5kdc[20795](info): commencing operation
krb5kdc: starting...
W20260927 11:16:53.310232 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:16:53 dist-test-slave-9hq9 krb5kdc[20795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:16:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:16:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:16:53.418939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37827
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:53.495850 20808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:53.496064 20808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:53.496111 20808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:53.497237 20808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:53.497304 20808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:53.497329 20808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:53.497359 20808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:53.497383 20808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:53.498834 20808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37827
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:53.499106 20808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:53.499315 20808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:53.503894 20814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:53.503901 20808 server_base.cc:1034] running on GCE node
W20260927 11:16:53.503863 20816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:53.503849 20813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:53.504323 20808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:53.504596 20808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:53.505759 20808 hybrid_clock.cc:648] HybridClock initialized: now 1790507813505759 us; error 36 us; skew 500 ppm
Sep 27 11:16:53 dist-test-slave-9hq9 krb5kdc[20795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:53.508011 20808 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:16:53.508395 20808 webserver.cc:460] Webserver started at http://127.0.0.1:38647/ using document root <none> and password file <none>
I20260927 11:16:53.508598 20808 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:53.508669 20808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:53.508806 20808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:53.509752 20808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "3bd8e1f83edf455fade18b6c35514c53"
format_stamp: "Formatted at 2026-09-27 11:16:53 on dist-test-slave-9hq9"
server_key: "0adbc28e00ccc6dbaff6a50f056b99e1"
server_key_iv: "37e25436514aa2a92b7a011cd923f0c7"
server_key_version: "encryptionkey@0"
I20260927 11:16:53.510180 20808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "3bd8e1f83edf455fade18b6c35514c53"
format_stamp: "Formatted at 2026-09-27 11:16:53 on dist-test-slave-9hq9"
server_key: "0adbc28e00ccc6dbaff6a50f056b99e1"
server_key_iv: "37e25436514aa2a92b7a011cd923f0c7"
server_key_version: "encryptionkey@0"
I20260927 11:16:53.511435 20808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:16:53.512123 20823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:53.512288 20808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:16:53.512383 20808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "3bd8e1f83edf455fade18b6c35514c53"
format_stamp: "Formatted at 2026-09-27 11:16:53 on dist-test-slave-9hq9"
server_key: "0adbc28e00ccc6dbaff6a50f056b99e1"
server_key_iv: "37e25436514aa2a92b7a011cd923f0c7"
server_key_version: "encryptionkey@0"
I20260927 11:16:53.512465 20808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:53.515610 20808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:53.517422 20808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:53.517568 20808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:53.521273 20808 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37827
I20260927 11:16:53.521318 20875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37827 every 8 connection(s)
I20260927 11:16:53.521615 20808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:53.522557 20876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:53.522790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 20808
I20260927 11:16:53.522924 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:16:53.523124 392 external_mini_cluster.cc:1428] Setting key 20f1e8a42ae6ecf185dc8f252f41b3cb
I20260927 11:16:53.525445 20876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53: Bootstrap starting.
I20260927 11:16:53.525616 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:53.523589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42561 (local address 127.0.0.1:37827)
0927 11:16:53.523797 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:53.523806 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:53.523837 (+ 31us) server_negotiation.cc:407] Connection header received
0927 11:16:53.524338 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:53.524345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:53.524376 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:53.524513 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:53.525153 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:53.525157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:53.525157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:53.525161 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:53.525201 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:53.525215 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:53.525294 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:53.525327 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 11:16:53.525455 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260927 11:16:53.526111 20876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:53.526435 20876 log.cc:824] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:53.526531 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:16:53.527068 20876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53: No bootstrap required, opened a new log
I20260927 11:16:53.528815 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:53.527940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36653 (local address 127.0.0.1:37827)
0927 11:16:53.527974 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:53.527976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:53.528030 (+ 54us) server_negotiation.cc:407] Connection header received
0927 11:16:53.528095 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:53.528095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:53.528099 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:53.528137 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:53.528608 (+ 471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:53.528609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:53.528609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:53.528613 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:53.528622 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:53.528626 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:53.528668 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:53.528732 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 11:16:53.528741 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:16:53.529315 20876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd8e1f83edf455fade18b6c35514c53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37827 } }
I20260927 11:16:53.529452 20876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:53.529495 20876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd8e1f83edf455fade18b6c35514c53, State: Initialized, Role: FOLLOWER
I20260927 11:16:53.529605 20876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd8e1f83edf455fade18b6c35514c53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37827 } }
I20260927 11:16:53.529711 20876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:53.529767 20876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:53.529812 20876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:53.530390 20876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd8e1f83edf455fade18b6c35514c53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37827 } }
I20260927 11:16:53.530501 20876 leader_election.cc:302] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bd8e1f83edf455fade18b6c35514c53; no voters:
I20260927 11:16:53.530680 20876 leader_election.cc:288] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:16:53.530723 20886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:53.530884 20886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [term 1 LEADER]: Becoming Leader. State: Replica: 3bd8e1f83edf455fade18b6c35514c53, State: Running, Role: LEADER
I20260927 11:16:53.530915 20876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:53.531018 20886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd8e1f83edf455fade18b6c35514c53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37827 } }
I20260927 11:16:53.531435 20887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bd8e1f83edf455fade18b6c35514c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd8e1f83edf455fade18b6c35514c53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37827 } } }
I20260927 11:16:53.531456 20888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bd8e1f83edf455fade18b6c35514c53. Latest consensus state: current_term: 1 leader_uuid: "3bd8e1f83edf455fade18b6c35514c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd8e1f83edf455fade18b6c35514c53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37827 } } }
I20260927 11:16:53.531548 20887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:53.531554 20888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:53.531741 20895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:53.532162 20895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:16:53.533649 20895 catalog_manager.cc:1347] Generated new cluster ID: 6be45f2fba8f4de68ab9e1289f4a47b2
I20260927 11:16:53.533703 20895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:16:53.544018 20895 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:16:53.544183 20895 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:53.551568 20895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bd8e1f83edf455fade18b6c35514c53: Generated new TSK 0
I20260927 11:16:53.551741 20895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:16:53.553484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 20808
2026-09-27T11:16:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1259 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:16:53 dist-test-slave-9hq9 krb5kdc[20907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:16:53 dist-test-slave-9hq9 krb5kdc[20907](info): set up 2 sockets
Sep 27 11:16:53 dist-test-slave-9hq9 krb5kdc[20907](info): commencing operation
krb5kdc: starting...
W20260927 11:16:54.577930 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:16:54 dist-test-slave-9hq9 krb5kdc[20907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:16:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:16:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:16:54.682922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:54.761543 20920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:54.761722 20920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:54.761744 20920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:54.762812 20920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:54.762848 20920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:54.762862 20920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:54.762874 20920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:54.762887 20920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:54.764256 20920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:54.764431 20920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:54.764577 20920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:16:54.769331 20920 server_base.cc:1034] running on GCE node
W20260927 11:16:54.769304 20925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:54.769290 20926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:54.769301 20928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:54.769840 20920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:54.770221 20920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:54.771375 20920 hybrid_clock.cc:648] HybridClock initialized: now 1790507814771361 us; error 32 us; skew 500 ppm
Sep 27 11:16:54 dist-test-slave-9hq9 krb5kdc[20907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:54.773571 20920 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:16:54.773998 20920 webserver.cc:460] Webserver started at http://127.0.0.1:37477/ using document root <none> and password file <none>
I20260927 11:16:54.774228 20920 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:54.774277 20920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:54.774359 20920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:54.775265 20920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "f268359374f94981b7b4453047a41578"
format_stamp: "Formatted at 2026-09-27 11:16:54 on dist-test-slave-9hq9"
server_key: "2152e0b8ead699455d07cc041d93beb4"
server_key_iv: "bf184c4f74f0ec0d1a0c813b2201fbd2"
server_key_version: "encryptionkey@0"
I20260927 11:16:54.775555 20920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "f268359374f94981b7b4453047a41578"
format_stamp: "Formatted at 2026-09-27 11:16:54 on dist-test-slave-9hq9"
server_key: "2152e0b8ead699455d07cc041d93beb4"
server_key_iv: "bf184c4f74f0ec0d1a0c813b2201fbd2"
server_key_version: "encryptionkey@0"
I20260927 11:16:54.776715 20920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:16:54.777292 20935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:54.777480 20920 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:54.777565 20920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "f268359374f94981b7b4453047a41578"
format_stamp: "Formatted at 2026-09-27 11:16:54 on dist-test-slave-9hq9"
server_key: "2152e0b8ead699455d07cc041d93beb4"
server_key_iv: "bf184c4f74f0ec0d1a0c813b2201fbd2"
server_key_version: "encryptionkey@0"
I20260927 11:16:54.777645 20920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:54.787556 20920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:54.789141 20920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:54.789289 20920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:54.793013 20920 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38157
I20260927 11:16:54.793041 20987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38157 every 8 connection(s)
I20260927 11:16:54.793366 20920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:54.794147 20988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:54.796491 20988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578: Bootstrap starting.
I20260927 11:16:54.796809 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 20920
I20260927 11:16:54.796936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:16:54.797115 20988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:54.797127 392 external_mini_cluster.cc:1428] Setting key 0b78ca92c0fcb36f772de62e37b9949e
I20260927 11:16:54.797426 20988 log.cc:824] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:54.798323 20988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578: No bootstrap required, opened a new log
I20260927 11:16:54.800486 20988 raft_consensus.cc:348] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f268359374f94981b7b4453047a41578" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38157 } }
I20260927 11:16:54.800635 20988 raft_consensus.cc:374] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:54.800668 20988 raft_consensus.cc:729] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f268359374f94981b7b4453047a41578, State: Initialized, Role: FOLLOWER
I20260927 11:16:54.800770 20988 consensus_queue.cc:260] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f268359374f94981b7b4453047a41578" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38157 } }
I20260927 11:16:54.800832 20988 raft_consensus.cc:388] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:54.800856 20988 raft_consensus.cc:482] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:54.800905 20988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:54.801487 20988 raft_consensus.cc:504] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f268359374f94981b7b4453047a41578" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38157 } }
I20260927 11:16:54.801570 20988 leader_election.cc:302] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f268359374f94981b7b4453047a41578; no voters:
I20260927 11:16:54.801765 20988 leader_election.cc:288] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:16:54.801810 20993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:54.801967 20993 raft_consensus.cc:686] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [term 1 LEADER]: Becoming Leader. State: Replica: f268359374f94981b7b4453047a41578, State: Running, Role: LEADER
I20260927 11:16:54.802093 20988 sys_catalog.cc:564] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:54.802098 20993 consensus_queue.cc:237] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f268359374f94981b7b4453047a41578" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38157 } }
I20260927 11:16:54.802593 20994 sys_catalog.cc:455] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f268359374f94981b7b4453047a41578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f268359374f94981b7b4453047a41578" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38157 } } }
I20260927 11:16:54.802584 20995 sys_catalog.cc:455] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f268359374f94981b7b4453047a41578. Latest consensus state: current_term: 1 leader_uuid: "f268359374f94981b7b4453047a41578" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f268359374f94981b7b4453047a41578" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38157 } } }
I20260927 11:16:54.802703 20994 sys_catalog.cc:458] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:54.802763 20995 sys_catalog.cc:458] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:54.803397 21002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:54.803893 21002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 11:16:54 dist-test-slave-9hq9 krb5kdc[20907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 11:16:54.805433 21002 catalog_manager.cc:1347] Generated new cluster ID: 5817cc44d3ec4bf6991995f55f2f572a
I20260927 11:16:54.805498 21002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:16:54.808014 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:54.797529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43611 (local address 127.0.0.1:38157)
0927 11:16:54.797694 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:54.797702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:54.797736 (+ 34us) server_negotiation.cc:407] Connection header received
0927 11:16:54.798263 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:54.798270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:54.798324 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:54.798488 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:54.799289 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:54.800133 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:54.801219 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:54.801644 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:54.801767 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:54.804612 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:54.804616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:54.804616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:54.804628 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:54.806504 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:54.806892 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:54.806894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:54.806895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:54.806983 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:54.807192 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:54.807192 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:54.807193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:54.807438 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:54.807569 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:54.807696 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 11:16:54.807820 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260927 11:16:54.808887 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:16:54.817032 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:54.810165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40427 (local address 127.0.0.1:38157)
0927 11:16:54.810216 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:54.810218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:54.810413 (+ 195us) server_negotiation.cc:407] Connection header received
0927 11:16:54.810440 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:54.810441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:54.810450 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:54.810513 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:54.811067 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:54.811770 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:54.812724 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:54.813046 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:54.813130 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:54.814350 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:54.814351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:54.814351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:54.814357 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:54.815926 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:54.816286 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:54.816287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:54.816287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:54.816340 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:54.816535 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:54.816536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:54.816536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:54.816649 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:54.816734 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:54.816871 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 11:16:54.816945 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 11:16:54.817287 21002 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:16:54.817852 21002 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:54.821170 21002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f268359374f94981b7b4453047a41578: Generated new TSK 0
I20260927 11:16:54.821321 21002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:16:54.824252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 20920
2026-09-27T11:16:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1270 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:16:54 dist-test-slave-9hq9 krb5kdc[21019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:16:54 dist-test-slave-9hq9 krb5kdc[21019](info): set up 2 sockets
Sep 27 11:16:54 dist-test-slave-9hq9 krb5kdc[21019](info): commencing operation
krb5kdc: starting...
W20260927 11:16:55.846182 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:16:55 dist-test-slave-9hq9 krb5kdc[21019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:16:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:16:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:16:55.960034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42743
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:56.039006 21032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:56.039191 21032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:56.039212 21032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:56.040323 21032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:56.040361 21032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:56.040376 21032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:56.040422 21032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:56.040449 21032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:56.041857 21032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:56.042095 21032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:56.042307 21032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:56.046798 21040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:56.046989 21032 server_base.cc:1034] running on GCE node
W20260927 11:16:56.046835 21038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:56.046814 21037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:56.047293 21032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:56.047518 21032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:56.048676 21032 hybrid_clock.cc:648] HybridClock initialized: now 1790507816048665 us; error 37 us; skew 500 ppm
Sep 27 11:16:56 dist-test-slave-9hq9 krb5kdc[21019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:56.051074 21032 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:16:56.051465 21032 webserver.cc:460] Webserver started at http://127.0.0.1:45487/ using document root <none> and password file <none>
I20260927 11:16:56.051679 21032 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:56.051744 21032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:56.051890 21032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:56.052826 21032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "2b982c4d61c04a57b94751592ed14fca"
format_stamp: "Formatted at 2026-09-27 11:16:56 on dist-test-slave-9hq9"
server_key: "0e4ce78baee8af97a6c5bed08bd08751"
server_key_iv: "1a13e980bbdaa290e0ee0ba4ee7d7087"
server_key_version: "encryptionkey@0"
I20260927 11:16:56.053170 21032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b982c4d61c04a57b94751592ed14fca"
format_stamp: "Formatted at 2026-09-27 11:16:56 on dist-test-slave-9hq9"
server_key: "0e4ce78baee8af97a6c5bed08bd08751"
server_key_iv: "1a13e980bbdaa290e0ee0ba4ee7d7087"
server_key_version: "encryptionkey@0"
I20260927 11:16:56.054479 21032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:16:56.055138 21047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:56.055289 21032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:56.055401 21032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "2b982c4d61c04a57b94751592ed14fca"
format_stamp: "Formatted at 2026-09-27 11:16:56 on dist-test-slave-9hq9"
server_key: "0e4ce78baee8af97a6c5bed08bd08751"
server_key_iv: "1a13e980bbdaa290e0ee0ba4ee7d7087"
server_key_version: "encryptionkey@0"
I20260927 11:16:56.055482 21032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:56.063074 21032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:56.065168 21032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:56.065312 21032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:56.069025 21032 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42743
I20260927 11:16:56.069069 21099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42743 every 8 connection(s)
I20260927 11:16:56.069384 21032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:56.070214 21100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:56.072163 21100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca: Bootstrap starting.
I20260927 11:16:56.072697 21100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:56.072964 21100 log.cc:824] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:56.073616 21100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca: No bootstrap required, opened a new log
I20260927 11:16:56.074434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 21032
I20260927 11:16:56.074571 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:16:56.074738 392 external_mini_cluster.cc:1428] Setting key 2466cda184c285bd8cef94faa1faad7b
I20260927 11:16:56.075704 21100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b982c4d61c04a57b94751592ed14fca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42743 } }
I20260927 11:16:56.075860 21100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:56.075898 21100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b982c4d61c04a57b94751592ed14fca, State: Initialized, Role: FOLLOWER
I20260927 11:16:56.076077 21100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b982c4d61c04a57b94751592ed14fca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42743 } }
I20260927 11:16:56.076164 21100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:56.076200 21100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:56.076253 21100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:56.076915 21100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b982c4d61c04a57b94751592ed14fca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42743 } }
I20260927 11:16:56.077011 21100 leader_election.cc:302] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b982c4d61c04a57b94751592ed14fca; no voters:
I20260927 11:16:56.077153 21100 leader_election.cc:288] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:16:56.077206 21105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:56.077353 21100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:56.077364 21105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [term 1 LEADER]: Becoming Leader. State: Replica: 2b982c4d61c04a57b94751592ed14fca, State: Running, Role: LEADER
I20260927 11:16:56.077493 21105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b982c4d61c04a57b94751592ed14fca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42743 } }
I20260927 11:16:56.078950 21106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b982c4d61c04a57b94751592ed14fca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b982c4d61c04a57b94751592ed14fca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42743 } } }
I20260927 11:16:56.077790 21105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b982c4d61c04a57b94751592ed14fca. Latest consensus state: current_term: 1 leader_uuid: "2b982c4d61c04a57b94751592ed14fca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b982c4d61c04a57b94751592ed14fca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42743 } } }
I20260927 11:16:56.079128 21106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:56.079138 21105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca [sys.catalog]: This master's current role is: LEADER
W20260927 11:16:56.079170 21120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:16:56.079242 21120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:16:56.079334 21122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:56.079481 21122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:16:56.080811 21122 catalog_manager.cc:1347] Generated new cluster ID: 7c196ff0a6ea449c953a06cdab7f91b3
I20260927 11:16:56.080857 21122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:16:56.081118 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:56.075226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33711 (local address 127.0.0.1:42743)
0927 11:16:56.075435 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:56.075444 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:56.075477 (+ 33us) server_negotiation.cc:407] Connection header received
0927 11:16:56.075670 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:56.075677 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:56.075730 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:56.075871 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:56.076122 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:56.077770 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:56.078983 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:56.079557 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:56.079665 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:56.080457 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:56.080460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:56.080460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:56.080655 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:56.080688 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:56.080701 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:56.080756 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:56.080931 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 11:16:56.080989 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":98,"threads_started":1}
I20260927 11:16:56.081878 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:16:56.089180 21122 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:16:56.089839 21122 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:56.091776 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:56.086285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33097 (local address 127.0.0.1:42743)
0927 11:16:56.086323 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:56.086325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:56.089042 (+ 2717us) server_negotiation.cc:407] Connection header received
0927 11:16:56.089115 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:56.089116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:56.089127 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:56.089184 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:56.089325 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:56.089766 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:56.090601 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:56.090792 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:56.090871 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:56.091544 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:56.091545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:56.091545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:56.091549 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:56.091561 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:56.091565 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:56.091611 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:56.091667 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 11:16:56.091699 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:16:56.094355 21122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b982c4d61c04a57b94751592ed14fca: Generated new TSK 0
I20260927 11:16:56.094493 21122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:16:56.098104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 21032
2026-09-27T11:16:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1273 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:16:56 dist-test-slave-9hq9 krb5kdc[21131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:16:56 dist-test-slave-9hq9 krb5kdc[21131](info): set up 2 sockets
Sep 27 11:16:56 dist-test-slave-9hq9 krb5kdc[21131](info): commencing operation
krb5kdc: starting...
W20260927 11:16:57.111048 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:16:57 dist-test-slave-9hq9 krb5kdc[21131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:16:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:16:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:16:57.222294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:57.298046 21144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:57.298390 21144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:57.298429 21144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:57.299576 21144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:57.299645 21144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:57.299679 21144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:57.299697 21144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:57.299724 21144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:57.301102 21144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:57.301354 21144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:57.301570 21144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:57.306093 21150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:57.306073 21149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:57.306087 21152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:57.306177 21144 server_base.cc:1034] running on GCE node
I20260927 11:16:57.306587 21144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:57.306814 21144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:57.308061 21144 hybrid_clock.cc:648] HybridClock initialized: now 1790507817308045 us; error 28 us; skew 500 ppm
Sep 27 11:16:57 dist-test-slave-9hq9 krb5kdc[21131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:57.313460 21144 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:16:57.313902 21144 webserver.cc:460] Webserver started at http://127.0.0.1:44339/ using document root <none> and password file <none>
I20260927 11:16:57.314175 21144 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:57.314219 21144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:57.314337 21144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:57.315318 21144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "2148aac8bacb43bb9dd0bb912b68225b"
format_stamp: "Formatted at 2026-09-27 11:16:57 on dist-test-slave-9hq9"
server_key: "4812558efc6bf82525241d9d5a206896"
server_key_iv: "08cef29e063d65e85ceb3278e5bdd961"
server_key_version: "encryptionkey@0"
I20260927 11:16:57.315652 21144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "2148aac8bacb43bb9dd0bb912b68225b"
format_stamp: "Formatted at 2026-09-27 11:16:57 on dist-test-slave-9hq9"
server_key: "4812558efc6bf82525241d9d5a206896"
server_key_iv: "08cef29e063d65e85ceb3278e5bdd961"
server_key_version: "encryptionkey@0"
I20260927 11:16:57.316919 21144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:16:57.317595 21159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:57.317783 21144 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:16:57.317858 21144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "2148aac8bacb43bb9dd0bb912b68225b"
format_stamp: "Formatted at 2026-09-27 11:16:57 on dist-test-slave-9hq9"
server_key: "4812558efc6bf82525241d9d5a206896"
server_key_iv: "08cef29e063d65e85ceb3278e5bdd961"
server_key_version: "encryptionkey@0"
I20260927 11:16:57.317907 21144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:57.320931 21144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:57.322654 21144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:57.322753 21144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:57.326292 21144 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46135
I20260927 11:16:57.326318 21211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46135 every 8 connection(s)
I20260927 11:16:57.326620 21144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:57.327512 21212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:57.329452 21212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b: Bootstrap starting.
I20260927 11:16:57.329952 21212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:57.330268 21212 log.cc:824] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:57.330869 21212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b: No bootstrap required, opened a new log
I20260927 11:16:57.332336 21212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2148aac8bacb43bb9dd0bb912b68225b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46135 } }
I20260927 11:16:57.332479 21212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:57.332522 21212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2148aac8bacb43bb9dd0bb912b68225b, State: Initialized, Role: FOLLOWER
I20260927 11:16:57.332666 21212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2148aac8bacb43bb9dd0bb912b68225b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46135 } }
I20260927 11:16:57.332746 21212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:57.332783 21212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:57.332834 21212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:57.333321 21212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2148aac8bacb43bb9dd0bb912b68225b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46135 } }
I20260927 11:16:57.333432 21212 leader_election.cc:302] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2148aac8bacb43bb9dd0bb912b68225b; no voters:
I20260927 11:16:57.333611 21212 leader_election.cc:288] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:16:57.333636 21215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:57.333806 21215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [term 1 LEADER]: Becoming Leader. State: Replica: 2148aac8bacb43bb9dd0bb912b68225b, State: Running, Role: LEADER
I20260927 11:16:57.333854 21212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:57.333945 21215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2148aac8bacb43bb9dd0bb912b68225b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46135 } }
I20260927 11:16:57.334296 21216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2148aac8bacb43bb9dd0bb912b68225b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2148aac8bacb43bb9dd0bb912b68225b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46135 } } }
I20260927 11:16:57.334388 21216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:57.334300 21217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2148aac8bacb43bb9dd0bb912b68225b. Latest consensus state: current_term: 1 leader_uuid: "2148aac8bacb43bb9dd0bb912b68225b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2148aac8bacb43bb9dd0bb912b68225b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46135 } } }
I20260927 11:16:57.334519 21217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:57.334667 21219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:57.335106 21219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:16:57.336602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 21144
I20260927 11:16:57.336658 21219 catalog_manager.cc:1347] Generated new cluster ID: a2cc003a891444bf82ba56810883a57d
I20260927 11:16:57.336705 21219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:16:57.336727 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:16:57.336905 392 external_mini_cluster.cc:1428] Setting key 62387fa4d641d20f0f0e37b7700a42bc
I20260927 11:16:57.339833 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:57.338173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53089 (local address 127.0.0.1:46135)
0927 11:16:57.338405 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:57.338413 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:57.338445 (+ 32us) server_negotiation.cc:407] Connection header received
0927 11:16:57.338563 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:57.338569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:57.338597 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:57.338738 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:57.339327 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:57.339331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:57.339331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:57.339337 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:57.339373 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:57.339388 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:57.339457 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:57.339533 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 11:16:57.339660 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260927 11:16:57.340678 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:16:57.347103 21235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:57.342129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34945 (local address 127.0.0.1:46135)
0927 11:16:57.342160 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:57.342162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:57.346222 (+ 4060us) server_negotiation.cc:407] Connection header received
0927 11:16:57.346308 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:57.346309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:57.346314 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:57.346365 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:57.346889 (+ 524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:57.346890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:57.346890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:57.346894 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:57.346905 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:57.346909 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:57.346977 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:57.347038 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 11:16:57.347050 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 11:16:57.349109 21219 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:16:57.349260 21219 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:57.363729 21219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2148aac8bacb43bb9dd0bb912b68225b: Generated new TSK 0
I20260927 11:16:57.363900 21219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:16:57.369642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 21144
2026-09-27T11:16:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:16:57 dist-test-slave-9hq9 krb5kdc[21243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:16:57 dist-test-slave-9hq9 krb5kdc[21243](info): set up 2 sockets
Sep 27 11:16:57 dist-test-slave-9hq9 krb5kdc[21243](info): commencing operation
krb5kdc: starting...
W20260927 11:16:58.390240 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:16:58 dist-test-slave-9hq9 krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:16:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:16:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:16:58.500519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45609
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:58.577560 21256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:58.577781 21256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:58.577817 21256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:58.578928 21256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:58.578992 21256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:58.579016 21256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:58.579043 21256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:58.579073 21256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:58.580463 21256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:58.580713 21256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:58.580914 21256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:58.585347 21262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:58.585343 21264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:58.585441 21256 server_base.cc:1034] running on GCE node
W20260927 11:16:58.585543 21261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:58.585767 21256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:58.586040 21256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:58.587213 21256 hybrid_clock.cc:648] HybridClock initialized: now 1790507818587202 us; error 25 us; skew 500 ppm
Sep 27 11:16:58 dist-test-slave-9hq9 krb5kdc[21243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:58.589422 21256 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:16:58.589838 21256 webserver.cc:460] Webserver started at http://127.0.0.1:43049/ using document root <none> and password file <none>
I20260927 11:16:58.590034 21256 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:58.590122 21256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:58.590261 21256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:58.591202 21256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "fdf3622b7b914f8f84ba616b306d4e76"
format_stamp: "Formatted at 2026-09-27 11:16:58 on dist-test-slave-9hq9"
server_key: "4d8862532a79d27161bbe112460a6090"
server_key_iv: "ba72cfd740706097b759bcfb15bff131"
server_key_version: "encryptionkey@0"
I20260927 11:16:58.591530 21256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "fdf3622b7b914f8f84ba616b306d4e76"
format_stamp: "Formatted at 2026-09-27 11:16:58 on dist-test-slave-9hq9"
server_key: "4d8862532a79d27161bbe112460a6090"
server_key_iv: "ba72cfd740706097b759bcfb15bff131"
server_key_version: "encryptionkey@0"
I20260927 11:16:58.592741 21256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:16:58.593366 21271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:58.593513 21256 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:16:58.593605 21256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "fdf3622b7b914f8f84ba616b306d4e76"
format_stamp: "Formatted at 2026-09-27 11:16:58 on dist-test-slave-9hq9"
server_key: "4d8862532a79d27161bbe112460a6090"
server_key_iv: "ba72cfd740706097b759bcfb15bff131"
server_key_version: "encryptionkey@0"
I20260927 11:16:58.593685 21256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:58.609001 21256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:58.610733 21256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:58.610874 21256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:58.614465 21256 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45609
I20260927 11:16:58.614518 21323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45609 every 8 connection(s)
I20260927 11:16:58.614850 21256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:58.615654 21324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:58.617653 21324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76: Bootstrap starting.
I20260927 11:16:58.618162 21324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:58.618387 21324 log.cc:824] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:58.618983 21324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76: No bootstrap required, opened a new log
I20260927 11:16:58.620419 21324 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf3622b7b914f8f84ba616b306d4e76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45609 } }
I20260927 11:16:58.620545 21324 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:58.620568 21324 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf3622b7b914f8f84ba616b306d4e76, State: Initialized, Role: FOLLOWER
I20260927 11:16:58.620667 21324 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf3622b7b914f8f84ba616b306d4e76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45609 } }
I20260927 11:16:58.620725 21324 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:58.620745 21324 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:58.620777 21324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:58.621212 21324 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf3622b7b914f8f84ba616b306d4e76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45609 } }
I20260927 11:16:58.621286 21324 leader_election.cc:302] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdf3622b7b914f8f84ba616b306d4e76; no voters:
I20260927 11:16:58.621397 21324 leader_election.cc:288] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:16:58.621436 21327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:58.621592 21327 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [term 1 LEADER]: Becoming Leader. State: Replica: fdf3622b7b914f8f84ba616b306d4e76, State: Running, Role: LEADER
I20260927 11:16:58.621621 21324 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:58.621743 21327 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf3622b7b914f8f84ba616b306d4e76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45609 } }
I20260927 11:16:58.622087 21328 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdf3622b7b914f8f84ba616b306d4e76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf3622b7b914f8f84ba616b306d4e76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45609 } } }
I20260927 11:16:58.622092 21329 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdf3622b7b914f8f84ba616b306d4e76. Latest consensus state: current_term: 1 leader_uuid: "fdf3622b7b914f8f84ba616b306d4e76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf3622b7b914f8f84ba616b306d4e76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45609 } } }
I20260927 11:16:58.622210 21328 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:58.622272 21329 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76 [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:58.622737 21331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:58.623229 21331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:16:58.624593 21331 catalog_manager.cc:1347] Generated new cluster ID: 372f37f38d7a4800ad29b32ea8e3fbab
I20260927 11:16:58.624588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 21256
I20260927 11:16:58.624643 21331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:16:58.624694 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:16:58.624898 392 external_mini_cluster.cc:1428] Setting key 67a248790053f85b4b91cb386c204aba
Sep 27 11:16:58 dist-test-slave-9hq9 krb5kdc[21243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 11:16:58.636346 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:58.625318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:35123 (local address 127.0.0.1:45609)
0927 11:16:58.625471 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:58.625480 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:58.625669 (+ 189us) server_negotiation.cc:407] Connection header received
0927 11:16:58.625809 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:58.625816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:58.625869 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:58.625985 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:58.626773 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:58.628148 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:58.629320 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:58.629904 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:58.629996 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:58.632737 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:58.632741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:58.632741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:58.632752 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:58.634732 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:58.635121 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:58.635124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:58.635125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:58.635231 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:58.635459 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:58.635460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:58.635460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:58.635758 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:58.635940 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:58.636019 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 11:16:58.636115 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260927 11:16:58.637632 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:16:58.637758 21331 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:16:58.638291 21331 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:58.646478 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:58.639111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:46585 (local address 127.0.0.1:45609)
0927 11:16:58.639159 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:58.639160 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:58.639351 (+ 191us) server_negotiation.cc:407] Connection header received
0927 11:16:58.639416 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:58.639417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:58.639426 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:58.639473 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:58.640043 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:58.640790 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:58.641869 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:58.642190 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:58.642244 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:58.643550 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:58.643551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:58.643551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:16:58.643557 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:58.645066 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:58.645465 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:58.645466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:58.645466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:58.645518 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:16:58.645772 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:16:58.645773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:16:58.645773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:16:58.645891 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:58.645986 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:58.646325 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 11:16:58.646389 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:16:58.648797 21331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdf3622b7b914f8f84ba616b306d4e76: Generated new TSK 0
I20260927 11:16:58.648980 21331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:16:58.653996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 21256
W20260927 11:16:58.658308 21241 connection.cc:535] client connection to 127.0.0.1:45609 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45609): Connection reset by peer (error 104)
2026-09-27T11:16:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:16:58 dist-test-slave-9hq9 krb5kdc[21355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:16:58 dist-test-slave-9hq9 krb5kdc[21355](info): set up 2 sockets
Sep 27 11:16:58 dist-test-slave-9hq9 krb5kdc[21355](info): commencing operation
krb5kdc: starting...
W20260927 11:16:59.677992 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:16:59 dist-test-slave-9hq9 krb5kdc[21355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:16:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:16:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:16:59.789677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41645
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:16:59.865991 21368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:16:59.866255 21368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:16:59.866295 21368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:16:59.867385 21368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:16:59.867444 21368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:16:59.867467 21368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:16:59.867496 21368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:16:59.867523 21368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:16:59.868873 21368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:16:59.869117 21368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:16:59.869315 21368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:16:59.873689 21373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:59.873701 21376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:16:59.873716 21374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:16:59.873895 21368 server_base.cc:1034] running on GCE node
I20260927 11:16:59.874174 21368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:16:59.874454 21368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:16:59.875597 21368 hybrid_clock.cc:648] HybridClock initialized: now 1790507819875587 us; error 26 us; skew 500 ppm
Sep 27 11:16:59 dist-test-slave-9hq9 krb5kdc[21355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:16:59.877902 21368 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:16:59.878350 21368 webserver.cc:460] Webserver started at http://127.0.0.1:44509/ using document root <none> and password file <none>
I20260927 11:16:59.878552 21368 fs_manager.cc:359] Metadata directory not provided
I20260927 11:16:59.878593 21368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:16:59.878731 21368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:16:59.879655 21368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "aadc296b41d942bf98ddb1da066a6f8c"
format_stamp: "Formatted at 2026-09-27 11:16:59 on dist-test-slave-9hq9"
server_key: "4655ba471416e19a3fbaff8e247e9042"
server_key_iv: "12fcf5d4bca8c467ab62eb2a3921fcc1"
server_key_version: "encryptionkey@0"
I20260927 11:16:59.880000 21368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "aadc296b41d942bf98ddb1da066a6f8c"
format_stamp: "Formatted at 2026-09-27 11:16:59 on dist-test-slave-9hq9"
server_key: "4655ba471416e19a3fbaff8e247e9042"
server_key_iv: "12fcf5d4bca8c467ab62eb2a3921fcc1"
server_key_version: "encryptionkey@0"
I20260927 11:16:59.881985 21368 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260927 11:16:59.882666 21383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:16:59.882835 21368 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:16:59.882941 21368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "aadc296b41d942bf98ddb1da066a6f8c"
format_stamp: "Formatted at 2026-09-27 11:16:59 on dist-test-slave-9hq9"
server_key: "4655ba471416e19a3fbaff8e247e9042"
server_key_iv: "12fcf5d4bca8c467ab62eb2a3921fcc1"
server_key_version: "encryptionkey@0"
I20260927 11:16:59.883013 21368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:16:59.891911 21368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:16:59.893457 21368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:16:59.893592 21368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:16:59.897416 21368 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41645
I20260927 11:16:59.897445 21435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41645 every 8 connection(s)
I20260927 11:16:59.897764 21368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:16:59.898530 21436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:16:59.900556 21436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c: Bootstrap starting.
I20260927 11:16:59.901087 21436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c: Neither blocks nor log segments found. Creating new log.
I20260927 11:16:59.901350 21436 log.cc:824] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c: Log is configured to *not* fsync() on all Append() calls
I20260927 11:16:59.901986 21436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c: No bootstrap required, opened a new log
I20260927 11:16:59.903646 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 21368
I20260927 11:16:59.903808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:16:59.903820 21436 raft_consensus.cc:348] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadc296b41d942bf98ddb1da066a6f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41645 } }
I20260927 11:16:59.903975 21436 raft_consensus.cc:374] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:16:59.904008 21436 raft_consensus.cc:729] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aadc296b41d942bf98ddb1da066a6f8c, State: Initialized, Role: FOLLOWER
I20260927 11:16:59.904039 392 external_mini_cluster.cc:1428] Setting key 6c7f906d3e3ccbb01590d5a40e54ba68
I20260927 11:16:59.904132 21436 consensus_queue.cc:260] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadc296b41d942bf98ddb1da066a6f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41645 } }
I20260927 11:16:59.904201 21436 raft_consensus.cc:388] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:16:59.904228 21436 raft_consensus.cc:482] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:16:59.904299 21436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:16:59.905369 21436 raft_consensus.cc:504] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadc296b41d942bf98ddb1da066a6f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41645 } }
I20260927 11:16:59.905474 21436 leader_election.cc:302] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aadc296b41d942bf98ddb1da066a6f8c; no voters:
I20260927 11:16:59.905623 21436 leader_election.cc:288] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:16:59.905669 21441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:16:59.905851 21441 raft_consensus.cc:686] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [term 1 LEADER]: Becoming Leader. State: Replica: aadc296b41d942bf98ddb1da066a6f8c, State: Running, Role: LEADER
I20260927 11:16:59.905938 21436 sys_catalog.cc:564] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:16:59.906044 21441 consensus_queue.cc:237] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadc296b41d942bf98ddb1da066a6f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41645 } }
I20260927 11:16:59.906522 21443 sys_catalog.cc:455] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aadc296b41d942bf98ddb1da066a6f8c. Latest consensus state: current_term: 1 leader_uuid: "aadc296b41d942bf98ddb1da066a6f8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadc296b41d942bf98ddb1da066a6f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41645 } } }
I20260927 11:16:59.906741 21443 sys_catalog.cc:458] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:59.906682 21442 sys_catalog.cc:455] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aadc296b41d942bf98ddb1da066a6f8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadc296b41d942bf98ddb1da066a6f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41645 } } }
I20260927 11:16:59.906972 21442 sys_catalog.cc:458] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c [sys.catalog]: This master's current role is: LEADER
I20260927 11:16:59.907112 21449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:16:59.907541 21449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:16:59.909071 21449 catalog_manager.cc:1347] Generated new cluster ID: 5e2f263ee1d34423b5cb1fab09d59c29
I20260927 11:16:59.909122 21449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:16:59.910768 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:59.904766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:59409 (local address 127.0.0.1:41645)
0927 11:16:59.905292 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:59.905301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:59.905338 (+ 37us) server_negotiation.cc:407] Connection header received
0927 11:16:59.905447 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:59.905454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:59.905514 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:59.905913 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:59.906418 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:59.907966 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:59.908855 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:59.909342 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:59.909692 (+ 350us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:59.910102 (+ 410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:59.910105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:59.910105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:59.910298 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:59.910329 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:59.910341 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:59.910402 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:59.910556 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 11:16:59.910578 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":100,"threads_started":1}
I20260927 11:16:59.911603 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:16:59.917433 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:16:59.912937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:58481 (local address 127.0.0.1:41645)
0927 11:16:59.912968 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 11:16:59.912969 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:16:59.914224 (+ 1255us) server_negotiation.cc:407] Connection header received
0927 11:16:59.914309 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:16:59.914310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:16:59.914321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:16:59.914395 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:16:59.914536 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:59.915256 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:59.916187 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:16:59.916501 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:16:59.916567 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:16:59.917190 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:16:59.917192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:16:59.917192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:16:59.917196 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:16:59.917209 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:16:59.917214 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:16:59.917277 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 11:16:59.917350 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 11:16:59.917362 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 11:16:59.928102 21449 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:16:59.928574 21449 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:16:59.943213 21449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aadc296b41d942bf98ddb1da066a6f8c: Generated new TSK 0
I20260927 11:16:59.943388 21449 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:16:59.945943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 21368
W20260927 11:16:59.950178 21353 connection.cc:535] client connection to 127.0.0.1:41645 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41645): Connection reset by peer (error 104)
2026-09-27T11:16:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:16:59 dist-test-slave-9hq9 krb5kdc[21467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:16:59 dist-test-slave-9hq9 krb5kdc[21467](info): set up 2 sockets
Sep 27 11:16:59 dist-test-slave-9hq9 krb5kdc[21467](info): commencing operation
krb5kdc: starting...
W20260927 11:17:00.961947 392 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:17:01 dist-test-slave-9hq9 krb5kdc[21467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:17:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:17:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:17:01.070794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:17:01.146734 21480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:01.146908 21480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:01.146929 21480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:01.148002 21480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:01.148036 21480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:01.148051 21480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:01.148062 21480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:01.148074 21480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:01.149410 21480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:01.149608 21480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:01.149755 21480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:17:01.154099 21485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:01.154116 21486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:01.154325 21480 server_base.cc:1034] running on GCE node
W20260927 11:17:01.154119 21488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:01.154665 21480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:01.154907 21480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:01.156065 21480 hybrid_clock.cc:648] HybridClock initialized: now 1790507821156048 us; error 28 us; skew 500 ppm
Sep 27 11:17:01 dist-test-slave-9hq9 krb5kdc[21467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:01.158301 21480 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:17:01.158689 21480 webserver.cc:460] Webserver started at http://127.0.0.1:38877/ using document root <none> and password file <none>
I20260927 11:17:01.158902 21480 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:01.158965 21480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:01.159101 21480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:01.160050 21480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "83a5dbb5b7df4ce88ce6ba3429e91e44"
format_stamp: "Formatted at 2026-09-27 11:17:01 on dist-test-slave-9hq9"
server_key: "d11a5b8ce9c3a14f3b8304a5d5014b7b"
server_key_iv: "c0087fe321ead3b5ca2d815724c13710"
server_key_version: "encryptionkey@0"
I20260927 11:17:01.160372 21480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "83a5dbb5b7df4ce88ce6ba3429e91e44"
format_stamp: "Formatted at 2026-09-27 11:17:01 on dist-test-slave-9hq9"
server_key: "d11a5b8ce9c3a14f3b8304a5d5014b7b"
server_key_iv: "c0087fe321ead3b5ca2d815724c13710"
server_key_version: "encryptionkey@0"
I20260927 11:17:01.161579 21480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 11:17:01.162293 21495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:01.162451 21480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:01.162539 21480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "83a5dbb5b7df4ce88ce6ba3429e91e44"
format_stamp: "Formatted at 2026-09-27 11:17:01 on dist-test-slave-9hq9"
server_key: "d11a5b8ce9c3a14f3b8304a5d5014b7b"
server_key_iv: "c0087fe321ead3b5ca2d815724c13710"
server_key_version: "encryptionkey@0"
I20260927 11:17:01.162607 21480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:01.165634 21480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:01.167317 21480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:01.167474 21480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:01.171038 21480 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36967
I20260927 11:17:01.171075 21547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36967 every 8 connection(s)
I20260927 11:17:01.171398 21480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:17:01.172272 21548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:01.174263 21548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44: Bootstrap starting.
I20260927 11:17:01.174785 21548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:01.174780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 21480
I20260927 11:17:01.174916 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:17:01.175091 392 external_mini_cluster.cc:1428] Setting key fb3071a6c3e98b6511a92e8fff2b6151
I20260927 11:17:01.175096 21548 log.cc:824] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:01.175793 21548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44: No bootstrap required, opened a new log
I20260927 11:17:01.177241 21552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:01.175520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:48123 (local address 127.0.0.1:36967)
0927 11:17:01.175670 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:01.175678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:01.175709 (+ 31us) server_negotiation.cc:407] Connection header received
0927 11:17:01.176121 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:01.176127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:01.176151 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:01.176254 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:01.176833 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:01.176835 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:01.176835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:17:01.176838 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:01.176865 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:17:01.176874 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:01.176939 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:01.177010 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 11:17:01.177065 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260927 11:17:01.178239 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:17:01.178352 21548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a5dbb5b7df4ce88ce6ba3429e91e44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36967 } }
I20260927 11:17:01.178498 21548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:17:01.178527 21548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a5dbb5b7df4ce88ce6ba3429e91e44, State: Initialized, Role: FOLLOWER
I20260927 11:17:01.178629 21548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a5dbb5b7df4ce88ce6ba3429e91e44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36967 } }
I20260927 11:17:01.178717 21548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:17:01.178753 21548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:17:01.178802 21548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:01.179499 21548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a5dbb5b7df4ce88ce6ba3429e91e44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36967 } }
I20260927 11:17:01.179610 21548 leader_election.cc:302] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83a5dbb5b7df4ce88ce6ba3429e91e44; no voters:
I20260927 11:17:01.179823 21548 leader_election.cc:288] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:17:01.179984 21559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:17:01.180065 21548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:01.180171 21559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [term 1 LEADER]: Becoming Leader. State: Replica: 83a5dbb5b7df4ce88ce6ba3429e91e44, State: Running, Role: LEADER
I20260927 11:17:01.180541 21559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a5dbb5b7df4ce88ce6ba3429e91e44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36967 } }
I20260927 11:17:01.180907 21552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:01.179731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:40127 (local address 127.0.0.1:36967)
0927 11:17:01.179781 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:01.179783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:01.179928 (+ 145us) server_negotiation.cc:407] Connection header received
0927 11:17:01.180003 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:01.180004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:01.180009 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:01.180065 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:01.180671 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:01.180672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:01.180673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:17:01.180678 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:01.180690 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:17:01.180694 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:01.180752 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:01.180826 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 11:17:01.180839 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 11:17:01.181162 21558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83a5dbb5b7df4ce88ce6ba3429e91e44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a5dbb5b7df4ce88ce6ba3429e91e44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36967 } } }
I20260927 11:17:01.181264 21558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:01.181452 21570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:17:01.181767 21560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83a5dbb5b7df4ce88ce6ba3429e91e44. Latest consensus state: current_term: 1 leader_uuid: "83a5dbb5b7df4ce88ce6ba3429e91e44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a5dbb5b7df4ce88ce6ba3429e91e44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36967 } } }
I20260927 11:17:01.181846 21560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:01.181988 21570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:17:01.183415 21570 catalog_manager.cc:1347] Generated new cluster ID: 87750c75999e4ec2968ae1275c3e1b3b
I20260927 11:17:01.183471 21570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:17:01.193194 21570 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:17:01.193332 21570 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:17:01.199034 21570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83a5dbb5b7df4ce88ce6ba3429e91e44: Generated new TSK 0
I20260927 11:17:01.199184 21570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:17:01.201601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 21480
2026-09-27T11:17:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1255 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:17:01 dist-test-slave-9hq9 krb5kdc[21579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:17:01 dist-test-slave-9hq9 krb5kdc[21579](info): set up 2 sockets
Sep 27 11:17:01 dist-test-slave-9hq9 krb5kdc[21579](info): commencing operation
krb5kdc: starting...
W20260927 11:17:02.218149 392 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:17:02 dist-test-slave-9hq9 krb5kdc[21579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:17:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:17:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:17:02.341550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42165
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:17:02.419135 21592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:02.419319 21592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:02.419341 21592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:02.420501 21592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:02.420540 21592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:02.420555 21592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:02.420568 21592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:02.420581 21592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:02.421958 21592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42165
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:02.422194 21592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:02.422362 21592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:17:02.426755 21597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:02.426750 21600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:02.426749 21598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:02.426965 21592 server_base.cc:1034] running on GCE node
I20260927 11:17:02.427229 21592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:02.427500 21592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:02.428647 21592 hybrid_clock.cc:648] HybridClock initialized: now 1790507822428639 us; error 32 us; skew 500 ppm
Sep 27 11:17:02 dist-test-slave-9hq9 krb5kdc[21579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:02.433687 21592 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:17:02.434191 21592 webserver.cc:460] Webserver started at http://127.0.0.1:42573/ using document root <none> and password file <none>
I20260927 11:17:02.434402 21592 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:02.434474 21592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:02.434604 21592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:02.435616 21592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "b19a7738ea7f49749b7c0cba44d05c92"
format_stamp: "Formatted at 2026-09-27 11:17:02 on dist-test-slave-9hq9"
server_key: "b8b2813ab71e76babad4dd2c7c00d318"
server_key_iv: "f3bf9699a7bf5442763f95c174a9e54e"
server_key_version: "encryptionkey@0"
I20260927 11:17:02.435958 21592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "b19a7738ea7f49749b7c0cba44d05c92"
format_stamp: "Formatted at 2026-09-27 11:17:02 on dist-test-slave-9hq9"
server_key: "b8b2813ab71e76babad4dd2c7c00d318"
server_key_iv: "f3bf9699a7bf5442763f95c174a9e54e"
server_key_version: "encryptionkey@0"
I20260927 11:17:02.437255 21592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:17:02.438016 21607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:02.438202 21592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:17:02.438294 21592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "b19a7738ea7f49749b7c0cba44d05c92"
format_stamp: "Formatted at 2026-09-27 11:17:02 on dist-test-slave-9hq9"
server_key: "b8b2813ab71e76babad4dd2c7c00d318"
server_key_iv: "f3bf9699a7bf5442763f95c174a9e54e"
server_key_version: "encryptionkey@0"
I20260927 11:17:02.438383 21592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:02.474923 21592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:02.476591 21592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:02.476739 21592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:02.480520 21592 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42165
I20260927 11:17:02.480552 21659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42165 every 8 connection(s)
I20260927 11:17:02.480885 21592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:17:02.481642 21660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:02.483734 21660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92: Bootstrap starting.
I20260927 11:17:02.484237 21660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:02.484457 21660 log.cc:824] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:02.485052 21660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92: No bootstrap required, opened a new log
I20260927 11:17:02.486207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 21592
I20260927 11:17:02.486315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:17:02.486497 392 external_mini_cluster.cc:1428] Setting key 9298ab109d345c9090fef706562af932
I20260927 11:17:02.486845 21660 raft_consensus.cc:348] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19a7738ea7f49749b7c0cba44d05c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42165 } }
I20260927 11:17:02.487011 21660 raft_consensus.cc:374] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:17:02.487074 21660 raft_consensus.cc:729] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b19a7738ea7f49749b7c0cba44d05c92, State: Initialized, Role: FOLLOWER
I20260927 11:17:02.487217 21660 consensus_queue.cc:260] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19a7738ea7f49749b7c0cba44d05c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42165 } }
I20260927 11:17:02.487298 21660 raft_consensus.cc:388] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:17:02.487326 21660 raft_consensus.cc:482] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:17:02.487370 21660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:02.488341 21660 raft_consensus.cc:504] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19a7738ea7f49749b7c0cba44d05c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42165 } }
I20260927 11:17:02.488442 21660 leader_election.cc:302] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b19a7738ea7f49749b7c0cba44d05c92; no voters:
I20260927 11:17:02.488591 21660 leader_election.cc:288] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:17:02.488651 21665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:17:02.488809 21665 raft_consensus.cc:686] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [term 1 LEADER]: Becoming Leader. State: Replica: b19a7738ea7f49749b7c0cba44d05c92, State: Running, Role: LEADER
I20260927 11:17:02.488811 21660 sys_catalog.cc:564] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:02.488935 21665 consensus_queue.cc:237] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19a7738ea7f49749b7c0cba44d05c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42165 } }
I20260927 11:17:02.489252 21665 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b19a7738ea7f49749b7c0cba44d05c92. Latest consensus state: current_term: 1 leader_uuid: "b19a7738ea7f49749b7c0cba44d05c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19a7738ea7f49749b7c0cba44d05c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42165 } } }
I20260927 11:17:02.489310 21665 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:02.489442 21666 sys_catalog.cc:455] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b19a7738ea7f49749b7c0cba44d05c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b19a7738ea7f49749b7c0cba44d05c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42165 } } }
I20260927 11:17:02.489549 21666 sys_catalog.cc:458] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92 [sys.catalog]: This master's current role is: LEADER
W20260927 11:17:02.490815 21680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:17:02.490869 21680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:17:02.490962 21681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:17:02.491099 21681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:17:02.492662 21681 catalog_manager.cc:1347] Generated new cluster ID: 3b3eb1c0970c46fca62e1f33fd1df9e7
I20260927 11:17:02.492712 21681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:17:02 dist-test-slave-9hq9 krb5kdc[21579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 11:17:02.497506 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:02.486967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:36689 (local address 127.0.0.1:42165)
0927 11:17:02.487152 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:02.487161 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:02.487215 (+ 54us) server_negotiation.cc:407] Connection header received
0927 11:17:02.487320 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:02.487326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:02.487383 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:02.487546 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:02.488091 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:02.489530 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:02.490683 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:02.491261 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:02.491366 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:02.494065 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:02.494087 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:02.494087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:02.494104 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:02.495933 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:02.496362 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:02.496365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:02.496365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:02.496473 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:02.496704 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:02.496705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:02.496705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:02.496974 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:02.497110 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:02.497228 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 11:17:02.497309 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260927 11:17:02.498395 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:17:02.506304 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:02.499749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:52547 (local address 127.0.0.1:42165)
0927 11:17:02.499790 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:02.499792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:02.499888 (+ 96us) server_negotiation.cc:407] Connection header received
0927 11:17:02.499959 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:02.499960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:02.499967 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:02.499991 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:02.500538 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:02.501233 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:02.502148 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:02.502442 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:02.502475 (+ 33us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:02.503703 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:02.503704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:02.503704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:02.503709 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:02.505078 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:02.505467 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:02.505468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:02.505469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:02.505515 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:02.505748 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:02.505749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:02.505749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:02.505847 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:02.505912 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:02.506156 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 11:17:02.506216 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:17:02.506592 21681 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:17:02.507174 21681 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:17:02.511005 21681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b19a7738ea7f49749b7c0cba44d05c92: Generated new TSK 0
I20260927 11:17:02.511149 21681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:17:02.513053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 21592
W20260927 11:17:02.517287 21577 connection.cc:535] client connection to 127.0.0.1:42165 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42165): Connection reset by peer (error 104)
2026-09-27T11:17:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:17:02 dist-test-slave-9hq9 krb5kdc[21691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:17:02 dist-test-slave-9hq9 krb5kdc[21691](info): set up 2 sockets
Sep 27 11:17:02 dist-test-slave-9hq9 krb5kdc[21691](info): commencing operation
krb5kdc: starting...
W20260927 11:17:03.522188 392 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:17:03 dist-test-slave-9hq9 krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:17:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:17:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:17:03.629837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:17:03.708132 21704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:03.708418 21704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:03.708458 21704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:03.710168 21704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:03.710228 21704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:03.710253 21704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:03.710275 21704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:03.710304 21704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:03.712083 21704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:03.712321 21704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:03.712545 21704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:17:03.716984 21710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:03.716989 21712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:03.717068 21704 server_base.cc:1034] running on GCE node
W20260927 11:17:03.716989 21709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:03.717396 21704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:03.717705 21704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:03.718851 21704 hybrid_clock.cc:648] HybridClock initialized: now 1790507823718829 us; error 34 us; skew 500 ppm
Sep 27 11:17:03 dist-test-slave-9hq9 krb5kdc[21691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:03.721107 21704 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:17:03.721477 21704 webserver.cc:460] Webserver started at http://127.0.0.1:45615/ using document root <none> and password file <none>
I20260927 11:17:03.721698 21704 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:03.721761 21704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:03.721897 21704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:03.722892 21704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "2fdf13d6a5684702af700814333880f4"
format_stamp: "Formatted at 2026-09-27 11:17:03 on dist-test-slave-9hq9"
server_key: "70ee1c94e6bc1bee8223170de815aa54"
server_key_iv: "ba72acd250d4ec95149808059b34fff0"
server_key_version: "encryptionkey@0"
I20260927 11:17:03.723240 21704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "2fdf13d6a5684702af700814333880f4"
format_stamp: "Formatted at 2026-09-27 11:17:03 on dist-test-slave-9hq9"
server_key: "70ee1c94e6bc1bee8223170de815aa54"
server_key_iv: "ba72acd250d4ec95149808059b34fff0"
server_key_version: "encryptionkey@0"
I20260927 11:17:03.724467 21704 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:17:03.725133 21719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:03.725292 21704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:17:03.725383 21704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "2fdf13d6a5684702af700814333880f4"
format_stamp: "Formatted at 2026-09-27 11:17:03 on dist-test-slave-9hq9"
server_key: "70ee1c94e6bc1bee8223170de815aa54"
server_key_iv: "ba72acd250d4ec95149808059b34fff0"
server_key_version: "encryptionkey@0"
I20260927 11:17:03.725461 21704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:03.736563 21704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:03.738236 21704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:03.738377 21704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:03.742430 21704 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43341
I20260927 11:17:03.742461 21771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43341 every 8 connection(s)
I20260927 11:17:03.742776 21704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:17:03.743687 21772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:03.743790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 21704
I20260927 11:17:03.743904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:17:03.744127 392 external_mini_cluster.cc:1428] Setting key 5ac436becc9631c4a8093d27c23f807e
I20260927 11:17:03.746547 21772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4: Bootstrap starting.
I20260927 11:17:03.747088 21772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:03.747354 21772 log.cc:824] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:03.747969 21772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4: No bootstrap required, opened a new log
I20260927 11:17:03.749359 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:03.744602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:38121 (local address 127.0.0.1:43341)
0927 11:17:03.744776 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:03.744785 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:03.744814 (+ 29us) server_negotiation.cc:407] Connection header received
0927 11:17:03.745228 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:03.745234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:03.745277 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:03.745411 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:03.745644 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:03.746576 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:03.747419 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:03.747836 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:03.747957 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:03.748616 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:03.748620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:03.748620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:17:03.748777 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:03.748807 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:17:03.748817 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:03.748905 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:03.749007 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 11:17:03.749035 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260927 11:17:03.749435 21772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdf13d6a5684702af700814333880f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } }
I20260927 11:17:03.749603 21772 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:17:03.749636 21772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fdf13d6a5684702af700814333880f4, State: Initialized, Role: FOLLOWER
I20260927 11:17:03.749758 21772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdf13d6a5684702af700814333880f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } }
I20260927 11:17:03.749861 21772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:17:03.749910 21772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:17:03.749960 21772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:03.750178 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:17:03.750566 21772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdf13d6a5684702af700814333880f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } }
I20260927 11:17:03.750675 21772 leader_election.cc:302] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fdf13d6a5684702af700814333880f4; no voters:
I20260927 11:17:03.750859 21772 leader_election.cc:288] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:17:03.750886 21777 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:17:03.750993 21777 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [term 1 LEADER]: Becoming Leader. State: Replica: 2fdf13d6a5684702af700814333880f4, State: Running, Role: LEADER
I20260927 11:17:03.751070 21777 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdf13d6a5684702af700814333880f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } }
I20260927 11:17:03.751099 21772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:03.751396 21778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fdf13d6a5684702af700814333880f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdf13d6a5684702af700814333880f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } } }
I20260927 11:17:03.751545 21778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:03.751883 21787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:17:03.752384 21787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:17:03.752816 21779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fdf13d6a5684702af700814333880f4. Latest consensus state: current_term: 1 leader_uuid: "2fdf13d6a5684702af700814333880f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fdf13d6a5684702af700814333880f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43341 } } }
I20260927 11:17:03.753754 21799 mvcc.cc:205] Tried to move back new op lower bound from 7333920046093053952 to 7333920046084726784. Current Snapshot: MvccSnapshot[applied={T|T < 7333920046093053952}]
I20260927 11:17:03.753767 21779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:03.754237 21787 catalog_manager.cc:1347] Generated new cluster ID: 2cd83052549242a18afb25560e7591ff
I20260927 11:17:03.754288 21787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:17:03.755049 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:03.751569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:48191 (local address 127.0.0.1:43341)
0927 11:17:03.751600 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:03.751602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:03.751714 (+ 112us) server_negotiation.cc:407] Connection header received
0927 11:17:03.751803 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:03.751804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:03.751816 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:03.751889 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:03.752096 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:03.752778 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:03.753792 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:03.754108 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:03.754170 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:03.754818 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:03.754819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:03.754819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:17:03.754823 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:03.754835 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:17:03.754839 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:03.754892 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:03.754949 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 11:17:03.754957 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:17:03.762117 21787 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:17:03.762588 21787 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:17:03.767397 21787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fdf13d6a5684702af700814333880f4: Generated new TSK 0
I20260927 11:17:03.767549 21787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:17:03.773257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 21704
W20260927 11:17:03.777222 21689 connection.cc:535] client connection to 127.0.0.1:43341 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43341): Connection reset by peer (error 104)
2026-09-27T11:17:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1259 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:17:03 dist-test-slave-9hq9 krb5kdc[21803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:17:03 dist-test-slave-9hq9 krb5kdc[21803](info): set up 2 sockets
Sep 27 11:17:03 dist-test-slave-9hq9 krb5kdc[21803](info): commencing operation
krb5kdc: starting...
W20260927 11:17:04.781975 392 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:17:04 dist-test-slave-9hq9 krb5kdc[21803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:17:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:17:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 11:17:04.888592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:17:04.967160 21816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:04.967394 21816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:04.967430 21816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:04.968534 21816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:04.968597 21816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:04.968619 21816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:04.968641 21816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:04.968667 21816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:04.970105 21816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:04.970364 21816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:04.970566 21816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:17:04.975039 21824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:04.975061 21822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:04.975061 21821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:04.975088 21816 server_base.cc:1034] running on GCE node
I20260927 11:17:04.975405 21816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:04.975683 21816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:04.976840 21816 hybrid_clock.cc:648] HybridClock initialized: now 1790507824976826 us; error 29 us; skew 500 ppm
Sep 27 11:17:04 dist-test-slave-9hq9 krb5kdc[21803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:04.979238 21816 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 11:17:04.979610 21816 webserver.cc:460] Webserver started at http://127.0.0.1:46017/ using document root <none> and password file <none>
I20260927 11:17:04.979828 21816 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:04.979873 21816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:04.980026 21816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:04.981011 21816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "2e37de8b75b3421baa08d25c76222785"
format_stamp: "Formatted at 2026-09-27 11:17:04 on dist-test-slave-9hq9"
server_key: "d22a54cf117dd68d308b30b73ea9dd52"
server_key_iv: "05f47704882f3dc4dd121b42f46ef75b"
server_key_version: "encryptionkey@0"
I20260927 11:17:04.981349 21816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e37de8b75b3421baa08d25c76222785"
format_stamp: "Formatted at 2026-09-27 11:17:04 on dist-test-slave-9hq9"
server_key: "d22a54cf117dd68d308b30b73ea9dd52"
server_key_iv: "05f47704882f3dc4dd121b42f46ef75b"
server_key_version: "encryptionkey@0"
I20260927 11:17:04.982671 21816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 11:17:04.983340 21831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:04.983502 21816 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:17:04.983593 21816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "2e37de8b75b3421baa08d25c76222785"
format_stamp: "Formatted at 2026-09-27 11:17:04 on dist-test-slave-9hq9"
server_key: "d22a54cf117dd68d308b30b73ea9dd52"
server_key_iv: "05f47704882f3dc4dd121b42f46ef75b"
server_key_version: "encryptionkey@0"
I20260927 11:17:04.983657 21816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:04.986832 21816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:04.988930 21816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:04.989028 21816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:04.992865 21816 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33201
I20260927 11:17:04.992926 21883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33201 every 8 connection(s)
I20260927 11:17:04.993209 21816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:17:04.994146 21884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:04.996083 21884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785: Bootstrap starting.
I20260927 11:17:04.996558 21884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:04.996778 21884 log.cc:824] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:04.997391 21884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785: No bootstrap required, opened a new log
I20260927 11:17:04.998909 21884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e37de8b75b3421baa08d25c76222785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } }
I20260927 11:17:04.999048 21884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:17:04.999069 21884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e37de8b75b3421baa08d25c76222785, State: Initialized, Role: FOLLOWER
I20260927 11:17:04.999197 21884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e37de8b75b3421baa08d25c76222785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } }
I20260927 11:17:04.999279 21884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:17:04.999300 21884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:17:04.999332 21884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:04.999804 21884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e37de8b75b3421baa08d25c76222785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } }
I20260927 11:17:04.999881 21884 leader_election.cc:302] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e37de8b75b3421baa08d25c76222785; no voters:
I20260927 11:17:04.999989 21884 leader_election.cc:288] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:17:05.000030 21887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:17:05.000173 21887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [term 1 LEADER]: Becoming Leader. State: Replica: 2e37de8b75b3421baa08d25c76222785, State: Running, Role: LEADER
I20260927 11:17:05.000237 21884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:05.000295 21887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e37de8b75b3421baa08d25c76222785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } }
I20260927 11:17:05.000651 21888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e37de8b75b3421baa08d25c76222785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e37de8b75b3421baa08d25c76222785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } } }
I20260927 11:17:05.000658 21889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e37de8b75b3421baa08d25c76222785. Latest consensus state: current_term: 1 leader_uuid: "2e37de8b75b3421baa08d25c76222785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e37de8b75b3421baa08d25c76222785" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33201 } } }
I20260927 11:17:05.000782 21888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:05.000857 21889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:05.001252 21891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:17:05.001796 21891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:17:05.002537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 21816
I20260927 11:17:05.002656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:17:05.002863 392 external_mini_cluster.cc:1428] Setting key f8007ee53b57fca71aa11a9d1483f778
I20260927 11:17:05.003420 21891 catalog_manager.cc:1347] Generated new cluster ID: 4df32d810ebe4381bc0061d56a88e0e9
I20260927 11:17:05.003471 21891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:17:05.013679 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:05.003227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:52099 (local address 127.0.0.1:33201)
0927 11:17:05.012228 (+ 9001us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:05.012237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:05.012270 (+ 33us) server_negotiation.cc:407] Connection header received
0927 11:17:05.012380 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:05.012387 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:05.012415 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:05.012551 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:05.013242 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:05.013246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:05.013246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:17:05.013251 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:05.013289 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:17:05.013302 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:05.013360 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:05.013450 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 11:17:05.013491 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8940,"thread_start_us":109,"threads_started":1}
I20260927 11:17:05.014513 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:17:05.015051 21891 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:17:05.015193 21891 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:17:05.016840 21907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:05.015919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.115:58077 (local address 127.0.0.1:33201)
0927 11:17:05.015948 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:05.015949 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:05.016057 (+ 108us) server_negotiation.cc:407] Connection header received
0927 11:17:05.016112 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:05.016112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:05.016117 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:05.016167 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:05.016653 (+ 486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:05.016654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:05.016655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:17:05.016658 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:05.016670 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:17:05.016674 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:05.016725 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:05.016769 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 11:17:05.016786 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 11:17:05.020231 21891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e37de8b75b3421baa08d25c76222785: Generated new TSK 0
I20260927 11:17:05.020385 21891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:17:05.023692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 21816
2026-09-27T11:17:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1250 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:17:05 dist-test-slave-9hq9 krb5kdc[21915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:17:05 dist-test-slave-9hq9 krb5kdc[21915](info): set up 2 sockets
Sep 27 11:17:05 dist-test-slave-9hq9 krb5kdc[21915](info): commencing operation
krb5kdc: starting...
W20260927 11:17:06.046022 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:17:06 dist-test-slave-9hq9 krb5kdc[21915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:17:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:17:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:17:06.152568 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41433
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:17:06.227327 21928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:06.227511 21928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:06.227545 21928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:06.228669 21928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:06.228729 21928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:06.228752 21928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:06.228798 21928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:06.228825 21928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:06.230159 21928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41433
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:06.230382 21928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:06.230567 21928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:17:06.234895 21934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:06.234941 21928 server_base.cc:1034] running on GCE node
W20260927 11:17:06.234908 21936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:06.234903 21933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:06.235342 21928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:06.235612 21928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:06.236763 21928 hybrid_clock.cc:648] HybridClock initialized: now 1790507826236756 us; error 33 us; skew 500 ppm
Sep 27 11:17:06 dist-test-slave-9hq9 krb5kdc[21915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:06.239094 21928 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:17:06.239488 21928 webserver.cc:460] Webserver started at http://127.0.98.62:34509/ using document root <none> and password file <none>
I20260927 11:17:06.239688 21928 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:06.239730 21928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:06.239867 21928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:06.240813 21928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "7d830e401cf04884a524b21607bbf066"
format_stamp: "Formatted at 2026-09-27 11:17:06 on dist-test-slave-9hq9"
server_key: "ef1b1f21a7eb4515519959825d870ae1"
server_key_iv: "2ac9b0e9f96278d5835a60eed7ba974f"
server_key_version: "encryptionkey@0"
I20260927 11:17:06.241142 21928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d830e401cf04884a524b21607bbf066"
format_stamp: "Formatted at 2026-09-27 11:17:06 on dist-test-slave-9hq9"
server_key: "ef1b1f21a7eb4515519959825d870ae1"
server_key_iv: "2ac9b0e9f96278d5835a60eed7ba974f"
server_key_version: "encryptionkey@0"
I20260927 11:17:06.242403 21928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 11:17:06.243155 21943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:06.243352 21928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:17:06.243443 21928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "7d830e401cf04884a524b21607bbf066"
format_stamp: "Formatted at 2026-09-27 11:17:06 on dist-test-slave-9hq9"
server_key: "ef1b1f21a7eb4515519959825d870ae1"
server_key_iv: "2ac9b0e9f96278d5835a60eed7ba974f"
server_key_version: "encryptionkey@0"
I20260927 11:17:06.243512 21928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:06.258667 21928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:06.260703 21928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:06.260846 21928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:06.264518 21928 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41433
I20260927 11:17:06.264569 21995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41433 every 8 connection(s)
I20260927 11:17:06.264853 21928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:17:06.265681 21996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:06.266471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 21928
I20260927 11:17:06.266601 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:17:06.266782 392 external_mini_cluster.cc:1428] Setting key c531350b8dc16f3f7bb373a877ad20cb
I20260927 11:17:06.268128 21996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066: Bootstrap starting.
I20260927 11:17:06.268864 21996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:06.269132 21996 log.cc:824] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:06.269767 21996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066: No bootstrap required, opened a new log
I20260927 11:17:06.272127 21996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d830e401cf04884a524b21607bbf066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41433 } }
I20260927 11:17:06.272295 21996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:17:06.272336 21996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d830e401cf04884a524b21607bbf066, State: Initialized, Role: FOLLOWER
I20260927 11:17:06.272464 21996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d830e401cf04884a524b21607bbf066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41433 } }
I20260927 11:17:06.272548 21996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:17:06.272578 21996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:17:06.272620 21996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:06.273175 21996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d830e401cf04884a524b21607bbf066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41433 } }
I20260927 11:17:06.273255 21996 leader_election.cc:302] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d830e401cf04884a524b21607bbf066; no voters:
Sep 27 11:17:06 dist-test-slave-9hq9 krb5kdc[21915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:17:06.273424 21996 leader_election.cc:288] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:17:06.273463 22001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:17:06.273644 22001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [term 1 LEADER]: Becoming Leader. State: Replica: 7d830e401cf04884a524b21607bbf066, State: Running, Role: LEADER
I20260927 11:17:06.273741 21996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:06.273767 22001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d830e401cf04884a524b21607bbf066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41433 } }
I20260927 11:17:06.274072 22003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d830e401cf04884a524b21607bbf066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d830e401cf04884a524b21607bbf066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41433 } } }
I20260927 11:17:06.274436 22003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:06.274060 22001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d830e401cf04884a524b21607bbf066. Latest consensus state: current_term: 1 leader_uuid: "7d830e401cf04884a524b21607bbf066" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d830e401cf04884a524b21607bbf066" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41433 } } }
I20260927 11:17:06.274752 22001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:06.274912 22010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:17:06.275439 22010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:17:06.276727 22010 catalog_manager.cc:1347] Generated new cluster ID: c63ae113a14f4529a56cd0d51d38f28a
I20260927 11:17:06.276777 22010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:17:06.277835 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:06.267249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45499 (local address 127.0.98.62:41433)
0927 11:17:06.267408 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:06.267422 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:06.267444 (+ 22us) server_negotiation.cc:407] Connection header received
0927 11:17:06.267965 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:06.267971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:06.268015 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:06.268121 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:06.268902 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:06.269820 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:06.270930 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:06.271293 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:06.271371 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:06.273850 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:06.273854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:06.273854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:06.273865 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:06.275875 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:06.276305 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:06.276308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:06.276308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:06.276425 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:06.276689 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:06.276691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:06.276691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:06.276963 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:06.277104 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:06.277540 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 11:17:06.277637 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260927 11:17:06.278699 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:17:06.286638 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:06.280119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51471 (local address 127.0.98.62:41433)
0927 11:17:06.280167 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:06.280169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:06.280231 (+ 62us) server_negotiation.cc:407] Connection header received
0927 11:17:06.280302 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:06.280303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:06.280312 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:06.280364 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:06.280907 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:06.281699 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:06.282612 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:06.282900 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:06.282972 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:06.284070 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:06.284071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:06.284071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:06.284077 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:06.285486 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:06.285855 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:06.285856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:06.285856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:06.285912 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:06.286129 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:06.286130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:06.286130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:06.286252 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:06.286342 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:06.286497 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 11:17:06.286555 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:17:06.287595 22010 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:17:06.288283 22010 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:17:06.292827 22010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d830e401cf04884a524b21607bbf066: Generated new TSK 0
I20260927 11:17:06.292968 22010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:17:06.297199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 21928
W20260927 11:17:06.300918 21913 connection.cc:535] client connection to 127.0.98.62:41433 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:41433): Connection reset by peer (error 104)
2026-09-27T11:17:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1273 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:17:06 dist-test-slave-9hq9 krb5kdc[22027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:17:06 dist-test-slave-9hq9 krb5kdc[22027](info): set up 2 sockets
Sep 27 11:17:06 dist-test-slave-9hq9 krb5kdc[22027](info): commencing operation
krb5kdc: starting...
W20260927 11:17:07.302066 392 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:17:07 dist-test-slave-9hq9 krb5kdc[22027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:17:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:17:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:17:07.410445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35583
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:17:07.488637 22040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:07.488824 22040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:07.488847 22040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:07.489894 22040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:07.489928 22040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:07.489941 22040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:07.489953 22040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:07.489965 22040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:07.491307 22040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35583
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:07.491473 22040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:07.491622 22040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:17:07.496259 22048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:07.496287 22045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:07.496531 22040 server_base.cc:1034] running on GCE node
W20260927 11:17:07.496259 22046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:07.496793 22040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:07.497069 22040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:07.498214 22040 hybrid_clock.cc:648] HybridClock initialized: now 1790507827498204 us; error 29 us; skew 500 ppm
Sep 27 11:17:07 dist-test-slave-9hq9 krb5kdc[22027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:07.505388 22040 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:17:07.505741 22040 webserver.cc:460] Webserver started at http://127.0.98.62:44355/ using document root <none> and password file <none>
I20260927 11:17:07.505934 22040 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:07.505990 22040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:07.506134 22040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:07.507081 22040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "54326bc86e0a42a490a9e1627943d276"
format_stamp: "Formatted at 2026-09-27 11:17:07 on dist-test-slave-9hq9"
server_key: "07dfca5dba7ae638cf9e56f75e17c6ba"
server_key_iv: "4443195c61677ca579fc1fd706d0a150"
server_key_version: "encryptionkey@0"
I20260927 11:17:07.507455 22040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "54326bc86e0a42a490a9e1627943d276"
format_stamp: "Formatted at 2026-09-27 11:17:07 on dist-test-slave-9hq9"
server_key: "07dfca5dba7ae638cf9e56f75e17c6ba"
server_key_iv: "4443195c61677ca579fc1fd706d0a150"
server_key_version: "encryptionkey@0"
I20260927 11:17:07.508679 22040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:17:07.509315 22055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:07.509485 22040 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:17:07.509573 22040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "54326bc86e0a42a490a9e1627943d276"
format_stamp: "Formatted at 2026-09-27 11:17:07 on dist-test-slave-9hq9"
server_key: "07dfca5dba7ae638cf9e56f75e17c6ba"
server_key_iv: "4443195c61677ca579fc1fd706d0a150"
server_key_version: "encryptionkey@0"
I20260927 11:17:07.509642 22040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:07.525831 22040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:07.527436 22040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:07.527581 22040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:07.531121 22040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35583
I20260927 11:17:07.531157 22107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35583 every 8 connection(s)
I20260927 11:17:07.531458 22040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:17:07.532226 22108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:07.534261 22108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276: Bootstrap starting.
I20260927 11:17:07.534749 22108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:07.534986 22108 log.cc:824] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:07.535187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 22040
I20260927 11:17:07.535321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:17:07.535502 392 external_mini_cluster.cc:1428] Setting key 2df5e0779050cc12e5b47cdd743dec90
I20260927 11:17:07.535759 22108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276: No bootstrap required, opened a new log
I20260927 11:17:07.538558 22108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54326bc86e0a42a490a9e1627943d276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } }
I20260927 11:17:07.538707 22108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:17:07.538734 22108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54326bc86e0a42a490a9e1627943d276, State: Initialized, Role: FOLLOWER
I20260927 11:17:07.538856 22108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54326bc86e0a42a490a9e1627943d276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } }
I20260927 11:17:07.538946 22108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:17:07.538981 22108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:17:07.539032 22108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:07.539683 22108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54326bc86e0a42a490a9e1627943d276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } }
I20260927 11:17:07.539780 22108 leader_election.cc:302] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54326bc86e0a42a490a9e1627943d276; no voters:
I20260927 11:17:07.539933 22108 leader_election.cc:288] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:17:07.539978 22113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:17:07.540138 22113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [term 1 LEADER]: Becoming Leader. State: Replica: 54326bc86e0a42a490a9e1627943d276, State: Running, Role: LEADER
I20260927 11:17:07.540180 22108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:07.540222 22113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54326bc86e0a42a490a9e1627943d276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } }
I20260927 11:17:07.540542 22114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54326bc86e0a42a490a9e1627943d276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54326bc86e0a42a490a9e1627943d276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } } }
I20260927 11:17:07.540627 22114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:07.540879 22115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54326bc86e0a42a490a9e1627943d276. Latest consensus state: current_term: 1 leader_uuid: "54326bc86e0a42a490a9e1627943d276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54326bc86e0a42a490a9e1627943d276" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } } }
I20260927 11:17:07.540987 22115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:07.541139 22119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:17:07.541640 22119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:17:07.542354 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:07.535946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44825 (local address 127.0.98.62:35583)
0927 11:17:07.536163 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:07.536172 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:07.536204 (+ 32us) server_negotiation.cc:407] Connection header received
0927 11:17:07.537023 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:07.537031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:07.537089 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:07.537223 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:07.537391 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:07.538837 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:07.540014 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:07.540652 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:07.540824 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:07.541508 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:07.541512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:07.541512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:17:07.541703 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:07.541739 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:17:07.541753 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:07.542098 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:07.542144 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 11:17:07.542169 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
I20260927 11:17:07.543157 22119 catalog_manager.cc:1347] Generated new cluster ID: 2b82a266cd3f4a389ebd8f8d6c8c97cf
I20260927 11:17:07.543206 22119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:17:07.543325 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:17:07.548377 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:07.544709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58873 (local address 127.0.98.62:35583)
0927 11:17:07.544735 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:07.544737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:07.544833 (+ 96us) server_negotiation.cc:407] Connection header received
0927 11:17:07.544919 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:07.544920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:07.544927 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:07.544950 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:07.545102 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:07.545731 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:07.546658 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:07.547390 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:07.547426 (+ 36us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:07.548139 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:07.548140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:07.548140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:17:07.548144 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:07.548154 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:17:07.548158 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:07.548186 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:07.548289 (+ 103us) server_negotiation.cc:299] Negotiation successful
0927 11:17:07.548297 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260927 11:17:07.552847 22119 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:17:07.553480 22119 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:17:07.568598 22119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54326bc86e0a42a490a9e1627943d276: Generated new TSK 0
I20260927 11:17:07.568791 22119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:17:07.574625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 22040
W20260927 11:17:07.578554 22025 connection.cc:535] client connection to 127.0.98.62:35583 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35583): Connection reset by peer (error 104)
2026-09-27T11:17:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:17:07 dist-test-slave-9hq9 krb5kdc[22139](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:17:07 dist-test-slave-9hq9 krb5kdc[22139](info): set up 2 sockets
Sep 27 11:17:07 dist-test-slave-9hq9 krb5kdc[22139](info): commencing operation
krb5kdc: starting...
W20260927 11:17:08.586109 392 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:17:08 dist-test-slave-9hq9 krb5kdc[22139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:17:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:17:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:17:08.693200 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39529
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:17:08.770521 22152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:08.770696 22152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:08.770717 22152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:08.771768 22152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:08.771799 22152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:08.771812 22152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:08.771826 22152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:08.771837 22152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:08.773170 22152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39529
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:08.773332 22152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:08.773488 22152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:17:08.777900 22157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:08.777858 22160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:08.777871 22158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:08.778107 22152 server_base.cc:1034] running on GCE node
I20260927 11:17:08.778331 22152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:08.778548 22152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:08.779709 22152 hybrid_clock.cc:648] HybridClock initialized: now 1790507828779699 us; error 39 us; skew 500 ppm
Sep 27 11:17:08 dist-test-slave-9hq9 krb5kdc[22139](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:08.782115 22152 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:17:08.782467 22152 webserver.cc:460] Webserver started at http://127.0.98.62:38819/ using document root <none> and password file <none>
I20260927 11:17:08.782624 22152 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:08.782661 22152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:08.782743 22152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:08.783651 22152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "7779ccd17c114244a3bb72b5b4c81fa3"
format_stamp: "Formatted at 2026-09-27 11:17:08 on dist-test-slave-9hq9"
server_key: "ec27d9092363e41b486297eb3020dd5f"
server_key_iv: "c2415e693ce2f0d558b2a1ddcfdac27d"
server_key_version: "encryptionkey@0"
I20260927 11:17:08.783931 22152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "7779ccd17c114244a3bb72b5b4c81fa3"
format_stamp: "Formatted at 2026-09-27 11:17:08 on dist-test-slave-9hq9"
server_key: "ec27d9092363e41b486297eb3020dd5f"
server_key_iv: "c2415e693ce2f0d558b2a1ddcfdac27d"
server_key_version: "encryptionkey@0"
I20260927 11:17:08.785058 22152 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:17:08.785662 22167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:08.785800 22152 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:17:08.785877 22152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "7779ccd17c114244a3bb72b5b4c81fa3"
format_stamp: "Formatted at 2026-09-27 11:17:08 on dist-test-slave-9hq9"
server_key: "ec27d9092363e41b486297eb3020dd5f"
server_key_iv: "c2415e693ce2f0d558b2a1ddcfdac27d"
server_key_version: "encryptionkey@0"
I20260927 11:17:08.785933 22152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:08.788919 22152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:08.790652 22152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:08.790751 22152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:08.794131 22152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39529
I20260927 11:17:08.794178 22219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39529 every 8 connection(s)
I20260927 11:17:08.794463 22152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:17:08.795362 22220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:08.797245 22220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3: Bootstrap starting.
I20260927 11:17:08.797501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 22152
I20260927 11:17:08.797639 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:17:08.797833 392 external_mini_cluster.cc:1428] Setting key c60df3230949ce316248bdc11a0af775
I20260927 11:17:08.797915 22220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:08.798353 22220 log.cc:824] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:08.799029 22220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3: No bootstrap required, opened a new log
I20260927 11:17:08.800258 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:08.798347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48619 (local address 127.0.98.62:39529)
0927 11:17:08.798555 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:08.798564 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:08.798597 (+ 33us) server_negotiation.cc:407] Connection header received
0927 11:17:08.799101 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:08.799110 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:08.799141 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:08.799262 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:08.799834 (+ 572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:08.799837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:08.799837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:17:08.799841 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:08.799873 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:17:08.799883 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:08.799949 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:08.800025 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 11:17:08.800071 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260927 11:17:08.801100 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:17:08.801438 22220 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7779ccd17c114244a3bb72b5b4c81fa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39529 } }
I20260927 11:17:08.801553 22220 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:17:08.801574 22220 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7779ccd17c114244a3bb72b5b4c81fa3, State: Initialized, Role: FOLLOWER
I20260927 11:17:08.801700 22220 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7779ccd17c114244a3bb72b5b4c81fa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39529 } }
I20260927 11:17:08.801783 22220 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:17:08.801805 22220 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:17:08.801870 22220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:08.802552 22220 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7779ccd17c114244a3bb72b5b4c81fa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39529 } }
I20260927 11:17:08.802706 22220 leader_election.cc:302] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7779ccd17c114244a3bb72b5b4c81fa3; no voters:
I20260927 11:17:08.802879 22220 leader_election.cc:288] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:17:08.802906 22230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:17:08.803045 22230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [term 1 LEADER]: Becoming Leader. State: Replica: 7779ccd17c114244a3bb72b5b4c81fa3, State: Running, Role: LEADER
I20260927 11:17:08.803113 22220 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:08.803133 22230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7779ccd17c114244a3bb72b5b4c81fa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39529 } }
I20260927 11:17:08.803459 22230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7779ccd17c114244a3bb72b5b4c81fa3. Latest consensus state: current_term: 1 leader_uuid: "7779ccd17c114244a3bb72b5b4c81fa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7779ccd17c114244a3bb72b5b4c81fa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39529 } } }
I20260927 11:17:08.803536 22230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:08.803694 22223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:08.802539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51421 (local address 127.0.98.62:39529)
0927 11:17:08.802575 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:08.802577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:08.802687 (+ 110us) server_negotiation.cc:407] Connection header received
0927 11:17:08.802729 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:08.802730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:08.802735 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:08.802796 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:08.803348 (+ 552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:08.803349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:08.803349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:17:08.803353 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:08.803365 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:17:08.803369 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:08.803410 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:08.803606 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 11:17:08.803619 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 11:17:08.803788 22231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7779ccd17c114244a3bb72b5b4c81fa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7779ccd17c114244a3bb72b5b4c81fa3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39529 } } }
I20260927 11:17:08.803854 22231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3 [sys.catalog]: This master's current role is: LEADER
W20260927 11:17:08.804728 22246 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:17:08.804790 22246 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:17:08.804833 22237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:17:08.804961 22237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:17:08.806313 22237 catalog_manager.cc:1347] Generated new cluster ID: 80f0c2790de342dd8a633364d4b6b439
I20260927 11:17:08.806360 22237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:17:08.814868 22237 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:17:08.815014 22237 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:17:08.819212 22237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7779ccd17c114244a3bb72b5b4c81fa3: Generated new TSK 0
I20260927 11:17:08.819350 22237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:17:08.824074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 22152
2026-09-27T11:17:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:17:08 dist-test-slave-9hq9 krb5kdc[22251](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:17:08 dist-test-slave-9hq9 krb5kdc[22251](info): set up 2 sockets
Sep 27 11:17:08 dist-test-slave-9hq9 krb5kdc[22251](info): commencing operation
krb5kdc: starting...
W20260927 11:17:09.842279 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:17:09 dist-test-slave-9hq9 krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:17:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:17:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:17:09.950750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45251
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:17:10.027889 22264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:10.028100 22264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:10.028136 22264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:10.029289 22264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:10.029373 22264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:10.029397 22264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:10.029418 22264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:10.029448 22264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:10.030995 22264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:10.031242 22264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:10.031446 22264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:17:10.035938 22270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:10.035957 22272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:10.035956 22269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:10.036087 22264 server_base.cc:1034] running on GCE node
I20260927 11:17:10.036314 22264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:10.036605 22264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:10.037774 22264 hybrid_clock.cc:648] HybridClock initialized: now 1790507830037776 us; error 42 us; skew 500 ppm
Sep 27 11:17:10 dist-test-slave-9hq9 krb5kdc[22251](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:10.040161 22264 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:17:10.040578 22264 webserver.cc:460] Webserver started at http://127.0.98.62:44815/ using document root <none> and password file <none>
I20260927 11:17:10.040771 22264 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:10.040817 22264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:10.040896 22264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:10.041893 22264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "e716dd5ca10c4b4da4e12bdc90038d9f"
format_stamp: "Formatted at 2026-09-27 11:17:10 on dist-test-slave-9hq9"
server_key: "e87e701894b5471f7d8d5e2b4e3a95f7"
server_key_iv: "e862f943472718d475fa287f1d6bb312"
server_key_version: "encryptionkey@0"
I20260927 11:17:10.042241 22264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "e716dd5ca10c4b4da4e12bdc90038d9f"
format_stamp: "Formatted at 2026-09-27 11:17:10 on dist-test-slave-9hq9"
server_key: "e87e701894b5471f7d8d5e2b4e3a95f7"
server_key_iv: "e862f943472718d475fa287f1d6bb312"
server_key_version: "encryptionkey@0"
I20260927 11:17:10.043412 22264 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 11:17:10.044044 22279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:10.044209 22264 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 11:17:10.044281 22264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "e716dd5ca10c4b4da4e12bdc90038d9f"
format_stamp: "Formatted at 2026-09-27 11:17:10 on dist-test-slave-9hq9"
server_key: "e87e701894b5471f7d8d5e2b4e3a95f7"
server_key_iv: "e862f943472718d475fa287f1d6bb312"
server_key_version: "encryptionkey@0"
I20260927 11:17:10.044332 22264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:10.069080 22264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:10.071277 22264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:10.071381 22264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:10.074936 22264 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45251
I20260927 11:17:10.074967 22331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45251 every 8 connection(s)
I20260927 11:17:10.075311 22264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:17:10.076126 22332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:10.078240 22332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f: Bootstrap starting.
I20260927 11:17:10.078786 22332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:10.079066 22332 log.cc:824] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:10.079712 22332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f: No bootstrap required, opened a new log
I20260927 11:17:10.081219 22332 raft_consensus.cc:348] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e716dd5ca10c4b4da4e12bdc90038d9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45251 } }
I20260927 11:17:10.081380 22332 raft_consensus.cc:374] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:17:10.081430 22332 raft_consensus.cc:729] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e716dd5ca10c4b4da4e12bdc90038d9f, State: Initialized, Role: FOLLOWER
I20260927 11:17:10.081619 22332 consensus_queue.cc:260] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e716dd5ca10c4b4da4e12bdc90038d9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45251 } }
I20260927 11:17:10.081730 22332 raft_consensus.cc:388] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:17:10.081775 22332 raft_consensus.cc:482] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:17:10.081827 22332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:10.082348 22332 raft_consensus.cc:504] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e716dd5ca10c4b4da4e12bdc90038d9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45251 } }
I20260927 11:17:10.082459 22332 leader_election.cc:302] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e716dd5ca10c4b4da4e12bdc90038d9f; no voters:
I20260927 11:17:10.082633 22332 leader_election.cc:288] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:17:10.082652 22335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:17:10.082804 22335 raft_consensus.cc:686] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [term 1 LEADER]: Becoming Leader. State: Replica: e716dd5ca10c4b4da4e12bdc90038d9f, State: Running, Role: LEADER
I20260927 11:17:10.082871 22332 sys_catalog.cc:564] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:10.082944 22335 consensus_queue.cc:237] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e716dd5ca10c4b4da4e12bdc90038d9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45251 } }
I20260927 11:17:10.083258 22337 sys_catalog.cc:455] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e716dd5ca10c4b4da4e12bdc90038d9f. Latest consensus state: current_term: 1 leader_uuid: "e716dd5ca10c4b4da4e12bdc90038d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e716dd5ca10c4b4da4e12bdc90038d9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45251 } } }
I20260927 11:17:10.083364 22336 sys_catalog.cc:455] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e716dd5ca10c4b4da4e12bdc90038d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e716dd5ca10c4b4da4e12bdc90038d9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45251 } } }
I20260927 11:17:10.083387 22337 sys_catalog.cc:458] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:10.083449 22336 sys_catalog.cc:458] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:10.083704 22344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:17:10.084266 22344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:17:10.085193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 22264
I20260927 11:17:10.085297 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:17:10.085484 392 external_mini_cluster.cc:1428] Setting key c2545a32be9f6d3557a774016410bfdd
I20260927 11:17:10.086037 22344 catalog_manager.cc:1347] Generated new cluster ID: 221b4aff62a6495588822f4c076bec31
I20260927 11:17:10.086109 22344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:17:10.095074 22344 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:17:10.095731 22344 catalog_manager.cc:1504] Loading token signing keys...
Sep 27 11:17:10 dist-test-slave-9hq9 krb5kdc[22251](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:17:10.099395 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:10.085931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45973 (local address 127.0.98.62:45251)
0927 11:17:10.090336 (+ 4405us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:10.090343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:10.090365 (+ 22us) server_negotiation.cc:407] Connection header received
0927 11:17:10.090448 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:10.090453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:10.090492 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:10.090622 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:10.091236 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:10.092460 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:10.093236 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:10.093661 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:10.093783 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:10.096235 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:10.096239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:10.096240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:10.096251 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:10.097859 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:10.098307 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:10.098310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:10.098310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:10.098411 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:10.098612 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:10.098613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:10.098613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:10.098816 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:10.098969 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:10.099068 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 11:17:10.099213 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4360,"thread_start_us":66,"threads_started":1}
I20260927 11:17:10.100178 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:17:10.103097 22344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e716dd5ca10c4b4da4e12bdc90038d9f: Generated new TSK 0
I20260927 11:17:10.103242 22344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:17:10.107954 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:10.101459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49395 (local address 127.0.98.62:45251)
0927 11:17:10.101497 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:10.101498 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:10.101614 (+ 116us) server_negotiation.cc:407] Connection header received
0927 11:17:10.101681 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:10.101682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:10.101705 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:10.101759 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:10.102297 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:10.102764 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:10.104107 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:10.104307 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:10.104363 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:10.105602 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:10.105603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:10.105603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:10.105609 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:10.106961 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:10.107333 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:10.107333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:10.107334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:10.107378 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:10.107561 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:10.107561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:10.107561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:10.107645 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:10.107712 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:10.107846 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 11:17:10.107902 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":19}
I20260927 11:17:10.109877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 22264
2026-09-27T11:17:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:17:10 dist-test-slave-9hq9 krb5kdc[22363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:17:10 dist-test-slave-9hq9 krb5kdc[22363](info): set up 2 sockets
Sep 27 11:17:10 dist-test-slave-9hq9 krb5kdc[22363](info): commencing operation
krb5kdc: starting...
W20260927 11:17:11.130306 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:17:11 dist-test-slave-9hq9 krb5kdc[22363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:17:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:17:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:17:11.236686 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37949
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:17:11.312883 22376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:11.313094 22376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:11.313128 22376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:11.314199 22376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:11.314261 22376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:11.314284 22376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:11.314311 22376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:11.314342 22376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:11.315780 22376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:11.316035 22376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:11.316241 22376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:17:11.320766 22384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:11.320789 22381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:11.320931 22382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:11.321081 22376 server_base.cc:1034] running on GCE node
I20260927 11:17:11.321345 22376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:11.321563 22376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:11.322714 22376 hybrid_clock.cc:648] HybridClock initialized: now 1790507831322698 us; error 42 us; skew 500 ppm
Sep 27 11:17:11 dist-test-slave-9hq9 krb5kdc[22363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:11.324883 22376 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:17:11.325248 22376 webserver.cc:460] Webserver started at http://127.0.98.62:37245/ using document root <none> and password file <none>
I20260927 11:17:11.325459 22376 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:11.325523 22376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:11.325661 22376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:11.326635 22376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "08430f95c8cd4387aba61a87adcb3663"
format_stamp: "Formatted at 2026-09-27 11:17:11 on dist-test-slave-9hq9"
server_key: "b25c5433e4c0c117589cc4fd94a8e448"
server_key_iv: "efd05ea712ea0cd6cb34c895dd136874"
server_key_version: "encryptionkey@0"
I20260927 11:17:11.326958 22376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "08430f95c8cd4387aba61a87adcb3663"
format_stamp: "Formatted at 2026-09-27 11:17:11 on dist-test-slave-9hq9"
server_key: "b25c5433e4c0c117589cc4fd94a8e448"
server_key_iv: "efd05ea712ea0cd6cb34c895dd136874"
server_key_version: "encryptionkey@0"
I20260927 11:17:11.328218 22376 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:17:11.328904 22391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:11.329094 22376 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:17:11.329195 22376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "08430f95c8cd4387aba61a87adcb3663"
format_stamp: "Formatted at 2026-09-27 11:17:11 on dist-test-slave-9hq9"
server_key: "b25c5433e4c0c117589cc4fd94a8e448"
server_key_iv: "efd05ea712ea0cd6cb34c895dd136874"
server_key_version: "encryptionkey@0"
I20260927 11:17:11.329273 22376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:11.343798 22376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:11.345376 22376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:11.345520 22376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:11.349054 22376 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37949
I20260927 11:17:11.349082 22443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37949 every 8 connection(s)
I20260927 11:17:11.349381 22376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:17:11.350221 22444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:11.350657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 22376
I20260927 11:17:11.350766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:17:11.350950 392 external_mini_cluster.cc:1428] Setting key 98767e19ceeaeb3d72b6eed7be82ce62
I20260927 11:17:11.353019 22444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663: Bootstrap starting.
I20260927 11:17:11.353633 22444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:11.353921 22444 log.cc:824] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:11.354554 22444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663: No bootstrap required, opened a new log
I20260927 11:17:11.356488 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:11.351399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33287 (local address 127.0.98.62:37949)
0927 11:17:11.351581 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:11.351590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:11.351622 (+ 32us) server_negotiation.cc:407] Connection header received
0927 11:17:11.352115 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:11.352123 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:11.352177 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:11.352335 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:11.352554 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:11.353886 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:11.354720 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:11.355096 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:11.355216 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:11.355895 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:11.355899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:11.355900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:17:11.356063 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:11.356088 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:17:11.356098 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:11.356163 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:11.356255 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 11:17:11.356333 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260927 11:17:11.356485 22444 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08430f95c8cd4387aba61a87adcb3663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37949 } }
I20260927 11:17:11.356638 22444 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:17:11.356668 22444 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08430f95c8cd4387aba61a87adcb3663, State: Initialized, Role: FOLLOWER
I20260927 11:17:11.356781 22444 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08430f95c8cd4387aba61a87adcb3663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37949 } }
I20260927 11:17:11.356864 22444 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:17:11.356894 22444 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:17:11.356941 22444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:11.357589 22444 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08430f95c8cd4387aba61a87adcb3663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37949 } }
I20260927 11:17:11.357618 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:17:11.357719 22444 leader_election.cc:302] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08430f95c8cd4387aba61a87adcb3663; no voters:
I20260927 11:17:11.357901 22444 leader_election.cc:288] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:17:11.357928 22449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:17:11.358038 22449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [term 1 LEADER]: Becoming Leader. State: Replica: 08430f95c8cd4387aba61a87adcb3663, State: Running, Role: LEADER
I20260927 11:17:11.358155 22444 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:11.358199 22449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08430f95c8cd4387aba61a87adcb3663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37949 } }
I20260927 11:17:11.358562 22449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08430f95c8cd4387aba61a87adcb3663. Latest consensus state: current_term: 1 leader_uuid: "08430f95c8cd4387aba61a87adcb3663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08430f95c8cd4387aba61a87adcb3663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37949 } } }
I20260927 11:17:11.358572 22450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08430f95c8cd4387aba61a87adcb3663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08430f95c8cd4387aba61a87adcb3663" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37949 } } }
I20260927 11:17:11.358724 22449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:11.358737 22450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:11.358994 22462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:17:11.359519 22462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:17:11.362717 22447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:11.359799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58589 (local address 127.0.98.62:37949)
0927 11:17:11.359829 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:11.359831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:11.360078 (+ 247us) server_negotiation.cc:407] Connection header received
0927 11:17:11.360139 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:11.360140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:11.360150 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:11.360204 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:11.360316 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:11.360767 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:11.361465 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:11.361670 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:11.361750 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:11.362421 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:11.362423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:11.362424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:17:11.362428 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:11.362443 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:17:11.362447 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:11.362504 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:11.362603 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 11:17:11.362624 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260927 11:17:11.363051 22462 catalog_manager.cc:1347] Generated new cluster ID: 0b9705ba66664c73a35b5caedc2bca8d
I20260927 11:17:11.363097 22462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:17:11.376262 22462 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:17:11.376739 22462 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:17:11.384106 22462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08430f95c8cd4387aba61a87adcb3663: Generated new TSK 0
I20260927 11:17:11.384263 22462 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:17:11.390973 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 22376
2026-09-27T11:17:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1279 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:17:11 dist-test-slave-9hq9 krb5kdc[22475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:17:11 dist-test-slave-9hq9 krb5kdc[22475](info): set up 2 sockets
Sep 27 11:17:11 dist-test-slave-9hq9 krb5kdc[22475](info): commencing operation
krb5kdc: starting...
W20260927 11:17:12.406116 392 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:17:12 dist-test-slave-9hq9 krb5kdc[22475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:17:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:17:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:17:12.514724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46341
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:17:12.591686 22488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:12.591876 22488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:12.591897 22488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:12.592911 22488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:12.592943 22488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:12.592957 22488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:12.592968 22488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:12.592980 22488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:12.594272 22488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:12.594427 22488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:12.594563 22488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:17:12.598868 22494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:12.598892 22493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:12.599010 22488 server_base.cc:1034] running on GCE node
W20260927 11:17:12.598889 22496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:12.599344 22488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:12.599609 22488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:12.600801 22488 hybrid_clock.cc:648] HybridClock initialized: now 1790507832600796 us; error 38 us; skew 500 ppm
Sep 27 11:17:12 dist-test-slave-9hq9 krb5kdc[22475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:12.616153 22488 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:17:12.616532 22488 webserver.cc:460] Webserver started at http://127.0.98.62:36305/ using document root <none> and password file <none>
I20260927 11:17:12.616731 22488 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:12.616801 22488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:12.616937 22488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:12.617864 22488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "790b72bbb6e8495c84c031217a2be0e5"
format_stamp: "Formatted at 2026-09-27 11:17:12 on dist-test-slave-9hq9"
server_key: "4a3360ce18f21c01166636df141fc2e6"
server_key_iv: "9459c42ba4bfd251cf58fcad0fbc5652"
server_key_version: "encryptionkey@0"
I20260927 11:17:12.618222 22488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "790b72bbb6e8495c84c031217a2be0e5"
format_stamp: "Formatted at 2026-09-27 11:17:12 on dist-test-slave-9hq9"
server_key: "4a3360ce18f21c01166636df141fc2e6"
server_key_iv: "9459c42ba4bfd251cf58fcad0fbc5652"
server_key_version: "encryptionkey@0"
I20260927 11:17:12.619499 22488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 11:17:12.620206 22503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:12.620415 22488 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:17:12.620507 22488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "790b72bbb6e8495c84c031217a2be0e5"
format_stamp: "Formatted at 2026-09-27 11:17:12 on dist-test-slave-9hq9"
server_key: "4a3360ce18f21c01166636df141fc2e6"
server_key_iv: "9459c42ba4bfd251cf58fcad0fbc5652"
server_key_version: "encryptionkey@0"
I20260927 11:17:12.620581 22488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:12.623808 22488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:12.625892 22488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:12.626039 22488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:12.629731 22488 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46341
I20260927 11:17:12.629765 22555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46341 every 8 connection(s)
I20260927 11:17:12.630131 22488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:17:12.631007 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:12.632964 22556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5: Bootstrap starting.
I20260927 11:17:12.633502 22556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:12.633769 22556 log.cc:824] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:12.634462 22556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5: No bootstrap required, opened a new log
I20260927 11:17:12.635891 22556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790b72bbb6e8495c84c031217a2be0e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46341 } }
I20260927 11:17:12.636011 22556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 11:17:12.636065 22556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 790b72bbb6e8495c84c031217a2be0e5, State: Initialized, Role: FOLLOWER
I20260927 11:17:12.636196 22556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790b72bbb6e8495c84c031217a2be0e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46341 } }
I20260927 11:17:12.636267 22556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 11:17:12.636287 22556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 11:17:12.636354 22556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:12.636829 22556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790b72bbb6e8495c84c031217a2be0e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46341 } }
I20260927 11:17:12.636927 22556 leader_election.cc:302] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 790b72bbb6e8495c84c031217a2be0e5; no voters:
I20260927 11:17:12.637087 22556 leader_election.cc:288] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 11:17:12.637241 22559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:17:12.637424 22559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 790b72bbb6e8495c84c031217a2be0e5, State: Running, Role: LEADER
I20260927 11:17:12.637463 22556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:12.637501 22559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790b72bbb6e8495c84c031217a2be0e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46341 } }
I20260927 11:17:12.638093 22561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 790b72bbb6e8495c84c031217a2be0e5. Latest consensus state: current_term: 1 leader_uuid: "790b72bbb6e8495c84c031217a2be0e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790b72bbb6e8495c84c031217a2be0e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46341 } } }
I20260927 11:17:12.638187 22561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:12.638329 22560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "790b72bbb6e8495c84c031217a2be0e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "790b72bbb6e8495c84c031217a2be0e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46341 } } }
I20260927 11:17:12.638415 22560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5 [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:12.638504 22568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:17:12.639027 22568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:17:12.639278 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 22488
I20260927 11:17:12.639392 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:17:12.639559 392 external_mini_cluster.cc:1428] Setting key 60194ae432d8362b3c4c1cf53e35e8cc
I20260927 11:17:12.641552 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:12.640116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60369 (local address 127.0.98.62:46341)
0927 11:17:12.640274 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:12.640282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:12.640310 (+ 28us) server_negotiation.cc:407] Connection header received
0927 11:17:12.640410 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:12.640417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:12.640443 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:12.640569 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:12.641181 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:12.641184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:12.641184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:17:12.641188 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:12.641223 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:17:12.641236 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:12.641317 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:12.641349 (+ 32us) server_negotiation.cc:299] Negotiation successful
0927 11:17:12.641385 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260927 11:17:12.642475 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:17:12.644903 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:12.643976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39459 (local address 127.0.98.62:46341)
0927 11:17:12.644029 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:12.644031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:12.644106 (+ 75us) server_negotiation.cc:407] Connection header received
0927 11:17:12.644189 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:12.644190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:12.644193 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:12.644238 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:12.644696 (+ 458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:12.644697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:12.644697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 11:17:12.644700 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:12.644710 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 11:17:12.644714 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:12.644754 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:12.644810 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 11:17:12.644820 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 11:17:12.646390 22568 catalog_manager.cc:1347] Generated new cluster ID: 83071b71ce224113b00767a84303cafc
I20260927 11:17:12.646448 22568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:17:12.664300 22568 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:17:12.664445 22568 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:17:12.669893 22568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 790b72bbb6e8495c84c031217a2be0e5: Generated new TSK 0
I20260927 11:17:12.670074 22568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 11:17:12.673254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 22488
2026-09-27T11:17:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1282 ms)
[----------] 18 tests from AuthTokenIssuingTest (22939 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:17:12 dist-test-slave-9hq9 krb5kdc[22587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:17:12 dist-test-slave-9hq9 krb5kdc[22587](info): set up 2 sockets
Sep 27 11:17:12 dist-test-slave-9hq9 krb5kdc[22587](info): commencing operation
krb5kdc: starting...
W20260927 11:17:13.682099 392 mini_kdc.cc:121] Time spent starting KDC: real 0.991s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:17:13 dist-test-slave-9hq9 krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:17:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:17:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:17:13.794348 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37523
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37523,127.0.98.61:40659,127.0.98.60:42017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:17:13.870721 22600 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 11:17:13.870945 22600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:13.870975 22600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:13.871017 22600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:13.872093 22600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:13.872156 22600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:13.872180 22600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:13.872210 22600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:13.872232 22600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:13.873785 22600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37523,127.0.98.61:40659,127.0.98.60:42017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:13.874039 22600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:13.874265 22600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:17:13.878585 22606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:13.878626 22605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:13.878723 22600 server_base.cc:1034] running on GCE node
W20260927 11:17:13.878775 22608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:13.878984 22600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:13.879282 22600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:13.880431 22600 hybrid_clock.cc:648] HybridClock initialized: now 1790507833880419 us; error 27 us; skew 500 ppm
Sep 27 11:17:13 dist-test-slave-9hq9 krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:13.882728 22600 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:17:13.883109 22600 webserver.cc:460] Webserver started at http://127.0.98.62:44667/ using document root <none> and password file <none>
I20260927 11:17:13.883280 22600 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:13.883319 22600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:13.883401 22600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:13.884286 22600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "e642c466bca941c49ae7bcc73036476c"
format_stamp: "Formatted at 2026-09-27 11:17:13 on dist-test-slave-9hq9"
server_key: "ecbf91bc5e1879fa98caa9581f713770"
server_key_iv: "442de0a40c2ba3fdca5d38a8a39fb261"
server_key_version: "encryptionkey@0"
I20260927 11:17:13.884560 22600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "e642c466bca941c49ae7bcc73036476c"
format_stamp: "Formatted at 2026-09-27 11:17:13 on dist-test-slave-9hq9"
server_key: "ecbf91bc5e1879fa98caa9581f713770"
server_key_iv: "442de0a40c2ba3fdca5d38a8a39fb261"
server_key_version: "encryptionkey@0"
I20260927 11:17:13.885715 22600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:17:13.886399 22615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:13.886543 22600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:13.886622 22600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "e642c466bca941c49ae7bcc73036476c"
format_stamp: "Formatted at 2026-09-27 11:17:13 on dist-test-slave-9hq9"
server_key: "ecbf91bc5e1879fa98caa9581f713770"
server_key_iv: "442de0a40c2ba3fdca5d38a8a39fb261"
server_key_version: "encryptionkey@0"
I20260927 11:17:13.886674 22600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:13.909605 22600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:13.911693 22600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:13.911794 22600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:13.915246 22600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37523
I20260927 11:17:13.915315 22667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37523 every 8 connection(s)
I20260927 11:17:13.915608 22600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:17:13.916502 22668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:13.917894 22668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:17:13.918486 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 22600
I20260927 11:17:13.918620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:17:13.918808 392 external_mini_cluster.cc:1428] Setting key c695bb96743253d0b2e08372355b1d5a
Sep 27 11:17:13 dist-test-slave-9hq9 krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260927 11:17:13.927102 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:13.918575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37523 (local address 127.0.0.1:34690)
0927 11:17:13.918740 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:13.918779 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:13.919237 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:13.919511 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:13.919514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:13.919797 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:13.920021 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:13.920026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:13.920984 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:13.920985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:13.921671 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:13.921673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:13.922239 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:13.922240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:13.922306 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:13.922786 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:13.922794 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:13.924060 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:13.925767 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:13.925786 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:13.925788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:13.925984 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:13.926247 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:13.926248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:13.926249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:13.926344 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:13.926697 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:13.926702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:13.926757 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 11:17:13.926832 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 11:17:13.926943 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260927 11:17:13.927107 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:13.918623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.0.98.62:37523)
0927 11:17:13.918815 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:13.918820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:13.918848 (+ 28us) server_negotiation.cc:407] Connection header received
0927 11:17:13.919386 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:13.919388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:13.919406 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:13.919507 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:13.920133 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:13.920901 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:13.921778 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:13.922134 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:13.922246 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:13.924154 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:13.924159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:13.924160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:13.924166 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:13.925651 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:13.926089 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:13.926092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:13.926093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:13.926179 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:13.926406 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:13.926407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:13.926408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:13.926599 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:13.926691 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:13.926867 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 11:17:13.926997 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260927 11:17:13.927731 22668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 11:17:13.928041 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:13.927946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40659 (local address 127.0.0.1:57706)
0927 11:17:13.927988 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:13.928025 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 11:17:13.928176 22618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40659: connect: Connection refused (error 111)
I20260927 11:17:13.928540 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:13.928462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40659 (local address 127.0.0.1:57720)
0927 11:17:13.928487 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:13.928512 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260927 11:17:13.928644 22668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40659: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40659: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 11:17:13.940878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40659
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37523,127.0.98.61:40659,127.0.98.60:42017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:17:13.977856 22668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } attempt: 1
I20260927 11:17:13.978351 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:13.978217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40659 (local address 127.0.0.1:57722)
0927 11:17:13.978273 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:13.978318 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 11:17:13.978696 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:13.978613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40659 (local address 127.0.0.1:57734)
0927 11:17:13.978653 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:13.978679 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 11:17:13.978816 22668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40659: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40659: connect: Connection refused (error 111)
W20260927 11:17:14.019243 22675 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 11:17:14.019433 22675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:14.019450 22675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:14.019469 22675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:14.020577 22675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:14.020615 22675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:14.020630 22675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:14.020643 22675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:14.020689 22675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:14.022087 22675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37523,127.0.98.61:40659,127.0.98.60:42017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:14.022311 22675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:14.022527 22675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:17:14.026928 22675 server_base.cc:1034] running on GCE node
W20260927 11:17:14.026909 22681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:14.026890 22683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:14.026902 22680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:14.027330 22675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:14.027583 22675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:14.028717 22675 hybrid_clock.cc:648] HybridClock initialized: now 1790507834028706 us; error 27 us; skew 500 ppm
Sep 27 11:17:14 dist-test-slave-9hq9 krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:14.031059 22675 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 11:17:14.031420 22675 webserver.cc:460] Webserver started at http://127.0.98.61:43579/ using document root <none> and password file <none>
I20260927 11:17:14.031576 22675 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:14.031612 22675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:14.031693 22675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:14.032596 22675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/data/instance:
uuid: "cff5b6c093f6461d89ca12664e0aa5f3"
format_stamp: "Formatted at 2026-09-27 11:17:14 on dist-test-slave-9hq9"
server_key: "0c8215352c758f9949e23f6c4f592ca0"
server_key_iv: "d96ced13f396a3239f7438bd2946d9c8"
server_key_version: "encryptionkey@0"
I20260927 11:17:14.032895 22675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/wal/instance:
uuid: "cff5b6c093f6461d89ca12664e0aa5f3"
format_stamp: "Formatted at 2026-09-27 11:17:14 on dist-test-slave-9hq9"
server_key: "0c8215352c758f9949e23f6c4f592ca0"
server_key_iv: "d96ced13f396a3239f7438bd2946d9c8"
server_key_version: "encryptionkey@0"
I20260927 11:17:14.034044 22675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:17:14.034766 22690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:14.034950 22675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:17:14.035043 22675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/wal
uuid: "cff5b6c093f6461d89ca12664e0aa5f3"
format_stamp: "Formatted at 2026-09-27 11:17:14 on dist-test-slave-9hq9"
server_key: "0c8215352c758f9949e23f6c4f592ca0"
server_key_iv: "d96ced13f396a3239f7438bd2946d9c8"
server_key_version: "encryptionkey@0"
I20260927 11:17:14.035127 22675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:14.046983 22668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } attempt: 2
I20260927 11:17:14.047292 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.047189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40659 (local address 127.0.0.1:57738)
0927 11:17:14.047239 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.047269 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 11:17:14.047578 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.047497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40659 (local address 127.0.0.1:57752)
0927 11:17:14.047535 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.047557 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260927 11:17:14.047648 22668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40659: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40659: connect: Connection refused (error 111)
I20260927 11:17:14.050175 22675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:14.051757 22675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:14.051914 22675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:14.055570 22675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40659
I20260927 11:17:14.055599 22742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40659 every 8 connection(s)
I20260927 11:17:14.055912 22675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/data/info.pb
I20260927 11:17:14.056756 22743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:14.058012 22743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:17:14 dist-test-slave-9hq9 krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:17:14.064803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 22675
I20260927 11:17:14.064881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-1/wal/instance
I20260927 11:17:14.065065 392 external_mini_cluster.cc:1428] Setting key 26a83f1f065fa5b363c815466573068a
I20260927 11:17:14.066373 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.058636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34706 (local address 127.0.98.62:37523)
0927 11:17:14.058688 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.058689 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.058871 (+ 182us) server_negotiation.cc:407] Connection header received
0927 11:17:14.059325 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.059327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.059338 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.059383 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.059900 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.060354 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.061299 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.061528 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.061630 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.063551 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:14.063552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:14.063553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:14.063559 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:14.065076 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.065464 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.065465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.065466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.065523 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.065744 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.065745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.065745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.065859 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:14.065922 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:14.066259 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 11:17:14.066322 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 11:17:14.066612 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.058614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37523 (local address 127.0.0.1:34706)
0927 11:17:14.058772 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.058801 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:14.059218 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:14.059440 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:14.059447 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:14.059649 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:14.059816 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.059820 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.060441 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.060442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.061195 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.061197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.061650 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.061651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.061767 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.062206 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:14.062213 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:14.063465 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:14.065168 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.065174 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.065175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.065384 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.065581 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.065582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.065582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.065681 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.065926 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:14.065932 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:14.066095 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 11:17:14.066217 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 11:17:14.066382 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260927 11:17:14.067109 22743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:17:14 dist-test-slave-9hq9 krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 11:17:14.074254 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.067343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40659 (local address 127.0.0.1:57756)
0927 11:17:14.067390 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.067401 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:14.067442 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:14.067643 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:14.067644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:14.067803 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:14.067857 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.067858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.068652 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.068654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.069536 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.069539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.069957 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.069959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.070042 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.070415 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:14.070421 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:14.071585 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:14.073402 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.073403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.073403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.073550 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.073719 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.073719 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.073720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.073765 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.074001 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:14.074001 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:14.074046 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 11:17:14.074132 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 11:17:14.074201 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 11:17:14.074362 22747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.067377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.0.98.61:40659)
0927 11:17:14.067508 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.067512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.067529 (+ 17us) server_negotiation.cc:407] Connection header received
0927 11:17:14.067549 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.067550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.067562 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.067625 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.067927 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.068577 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.069615 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.069886 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.069934 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.071666 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:14.071675 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:14.071675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:14.071682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:14.073317 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.073618 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.073620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.073620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.073667 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.073801 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.073801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.073801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.073929 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:14.074001 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:14.074155 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 11:17:14.074248 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":106,"thread_start_us":85,"threads_started":1}
I20260927 11:17:14.074831 22743 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:17:14.075155 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.075005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42017 (local address 127.0.0.1:42020)
0927 11:17:14.075066 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.075136 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
W20260927 11:17:14.075274 22693 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
I20260927 11:17:14.075628 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.075545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42017 (local address 127.0.0.1:42022)
0927 11:17:14.075587 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.075609 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 11:17:14.075749 22743 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42017: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 11:17:14.088137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42017
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37523,127.0.98.61:40659,127.0.98.60:42017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:17:14.125010 22743 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } attempt: 1
I20260927 11:17:14.125480 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.125350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42017 (local address 127.0.0.1:42032)
0927 11:17:14.125422 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.125457 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 11:17:14.125851 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.125760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42017 (local address 127.0.0.1:42038)
0927 11:17:14.125807 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.125830 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 11:17:14.125945 22743 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42017: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
I20260927 11:17:14.138767 22668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } attempt: 3
Sep 27 11:17:14 dist-test-slave-9hq9 krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:17:14.145308 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.139025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40659 (local address 127.0.0.1:57760)
0927 11:17:14.139073 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.139085 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:14.139149 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:14.139279 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:14.139279 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:14.139423 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:14.139482 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.139483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.140138 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.140139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.140660 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.140661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.141047 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.141048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.141129 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.141579 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:14.141583 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:14.142752 (+ 1169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:14.144316 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.144317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.144318 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.144487 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.144692 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.144692 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.144693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.144749 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.144999 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:14.145000 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:14.145043 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 11:17:14.145123 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 11:17:14.145216 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 11:17:14.145304 22747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.139095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.0.98.61:40659)
0927 11:17:14.139155 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.139157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.139182 (+ 25us) server_negotiation.cc:407] Connection header received
0927 11:17:14.139205 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.139206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.139220 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.139273 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.139562 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.140052 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.140766 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.140978 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.141050 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.142835 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:14.142836 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:14.142837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:14.142844 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:14.144217 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.144568 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.144569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.144570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.144626 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.144823 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.144824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.144824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.144928 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:14.144996 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:14.145146 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 11:17:14.145215 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 11:17:14.145623 22668 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:17:14.145937 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.145841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42017 (local address 127.0.0.1:42052)
0927 11:17:14.145890 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.145915 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 11:17:14.146266 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.146178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42017 (local address 127.0.0.1:42056)
0927 11:17:14.146223 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.146245 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 11:17:14.146384 22668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42017: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
W20260927 11:17:14.165812 22750 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 11:17:14.165988 22750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:14.166005 22750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:14.166080 22750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:14.167201 22750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:14.167240 22750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:14.167255 22750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:14.167315 22750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:14.167364 22750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:14.168769 22750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37523,127.0.98.61:40659,127.0.98.60:42017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:14.168995 22750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:14.169206 22750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:17:14.173727 22756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:14.173760 22755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:14.173734 22758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:14.173974 22750 server_base.cc:1034] running on GCE node
I20260927 11:17:14.174325 22750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:14.174597 22750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:14.175758 22750 hybrid_clock.cc:648] HybridClock initialized: now 1790507834175752 us; error 31 us; skew 500 ppm
I20260927 11:17:14.177518 22668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } attempt: 1
Sep 27 11:17:14 dist-test-slave-9hq9 krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:14.177891 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.177755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42017 (local address 127.0.0.1:42064)
0927 11:17:14.177815 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.177855 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 11:17:14.178164 22750 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 11:17:14.178234 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.178159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42017 (local address 127.0.0.1:42072)
0927 11:17:14.178189 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.178213 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260927 11:17:14.178344 22668 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42017: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
I20260927 11:17:14.178640 22750 webserver.cc:460] Webserver started at http://127.0.98.60:42487/ using document root <none> and password file <none>
I20260927 11:17:14.178838 22750 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:14.178886 22750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:14.179020 22750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:14.180009 22750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/data/instance:
uuid: "ce01f6045504425f857a6bc84021cfc9"
format_stamp: "Formatted at 2026-09-27 11:17:14 on dist-test-slave-9hq9"
server_key: "5fa3f048167d494a1e90ad347c86406d"
server_key_iv: "1c98f6f2051a3c6b4163c48c6a4bcda8"
server_key_version: "encryptionkey@0"
I20260927 11:17:14.180341 22750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/wal/instance:
uuid: "ce01f6045504425f857a6bc84021cfc9"
format_stamp: "Formatted at 2026-09-27 11:17:14 on dist-test-slave-9hq9"
server_key: "5fa3f048167d494a1e90ad347c86406d"
server_key_iv: "1c98f6f2051a3c6b4163c48c6a4bcda8"
server_key_version: "encryptionkey@0"
I20260927 11:17:14.181552 22750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:17:14.182286 22765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:14.182449 22750 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:14.182531 22750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/wal
uuid: "ce01f6045504425f857a6bc84021cfc9"
format_stamp: "Formatted at 2026-09-27 11:17:14 on dist-test-slave-9hq9"
server_key: "5fa3f048167d494a1e90ad347c86406d"
server_key_iv: "1c98f6f2051a3c6b4163c48c6a4bcda8"
server_key_version: "encryptionkey@0"
I20260927 11:17:14.182598 22750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:14.194169 22743 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } attempt: 2
I20260927 11:17:14.194519 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.194410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42017 (local address 127.0.0.1:42078)
0927 11:17:14.194466 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.194499 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 11:17:14.194864 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.194779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42017 (local address 127.0.0.1:42088)
0927 11:17:14.194823 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.194848 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 11:17:14.194965 22743 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42017: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42017: connect: Connection refused (error 111)
I20260927 11:17:14.196676 22750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:14.198724 22750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:14.198822 22750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:14.202418 22750 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42017
I20260927 11:17:14.202445 22817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42017 every 8 connection(s)
I20260927 11:17:14.202745 22750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/data/info.pb
I20260927 11:17:14.203656 22818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:14.205029 22818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:17:14 dist-test-slave-9hq9 krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:17:14.212160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 22750
I20260927 11:17:14.212265 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790507576023819-392-0/minicluster-data/master-2/wal/instance
I20260927 11:17:14.212440 392 external_mini_cluster.cc:1428] Setting key 7589da623c57636034ba871e56ac6a47
I20260927 11:17:14.212811 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.205620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34722 (local address 127.0.98.62:37523)
0927 11:17:14.205682 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.205684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.205895 (+ 211us) server_negotiation.cc:407] Connection header received
0927 11:17:14.206364 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.206365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.206375 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.206427 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.206916 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.207368 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.208208 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.208394 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.208437 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.210195 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:14.210196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:14.210197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:14.210202 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:14.211509 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.211821 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.211821 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.211821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.211868 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.212150 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.212151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.212151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.212240 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:14.212323 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:14.212641 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 11:17:14.212721 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 11:17:14.212975 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.205622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37523 (local address 127.0.0.1:34722)
0927 11:17:14.205775 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.205812 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:14.206281 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:14.206484 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:14.206486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:14.206685 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:14.206852 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.206856 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.207441 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.207442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.208147 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.208149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.208464 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.208465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.208570 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.208910 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:14.208917 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:14.210125 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:14.211572 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.211576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.211577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.211761 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.211930 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.211930 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.211930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.212078 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.212318 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:14.212324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:14.212472 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 11:17:14.212606 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 11:17:14.212752 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260927 11:17:14.213450 22818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:17:14 dist-test-slave-9hq9 krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 11:17:14 dist-test-slave-9hq9 krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:17:14.220403 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.213690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40659 (local address 127.0.0.1:57774)
0927 11:17:14.213803 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.213814 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:14.213868 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:14.214023 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:14.214023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:14.214169 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:14.214208 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.214209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.215033 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.215034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.215792 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.215794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.216199 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.216200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.216243 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.216623 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:14.216639 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:14.217781 (+ 1142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:14.219312 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.219313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.219314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.219478 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.219701 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.219701 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.219702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.219763 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.220070 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:14.220071 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:14.220116 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 11:17:14.220228 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 11:17:14.220317 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 11:17:14.220412 22747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.213757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.0.98.61:40659)
0927 11:17:14.213807 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.213809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.213854 (+ 45us) server_negotiation.cc:407] Connection header received
0927 11:17:14.213948 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.213948 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.213958 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.214020 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.214276 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.214966 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.215868 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.216129 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.216195 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.217869 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:14.217871 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:14.217878 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:14.217885 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:14.219234 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.219561 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.219562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.219563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.219613 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.219823 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.219824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.219824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.219933 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:14.220082 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:14.220259 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 11:17:14.220332 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 11:17:14.220762 22818 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:17:14.221421 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.212813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.0.98.60:42017)
0927 11:17:14.212959 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.212962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.213135 (+ 173us) server_negotiation.cc:407] Connection header received
0927 11:17:14.213158 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.213160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.213172 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.213232 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.213944 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.214956 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.215955 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.216357 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.216435 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.218567 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:14.218572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:14.218573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:14.218596 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:14.220048 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.220432 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.220435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.220435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.220614 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.220820 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.220821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.220821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.220987 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:14.221127 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:14.221228 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 11:17:14.221301 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
Sep 27 11:17:14 dist-test-slave-9hq9 krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 11:17:14 dist-test-slave-9hq9 krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:17:14.228024 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.220987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42017 (local address 127.0.0.1:42104)
0927 11:17:14.221045 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.221055 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:14.221088 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:14.221289 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:14.221290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:14.221449 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:14.221469 (+ 20us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.221470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.222304 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.222305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.222890 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.222893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.223235 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.223236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.223295 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.223730 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:14.223735 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:14.224841 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:14.226971 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.226972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.226972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.227097 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.227303 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.227304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.227304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.227358 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.227617 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:14.227618 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:14.227681 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 11:17:14.227749 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 11:17:14.227843 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 11:17:14.228219 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.221047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.0.98.60:42017)
0927 11:17:14.221172 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.221174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.221179 (+ 5us) server_negotiation.cc:407] Connection header received
0927 11:17:14.221195 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.221195 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.221205 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.221287 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.221561 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.222214 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.222971 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.223156 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.223214 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.224917 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:14.224918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:14.224919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:14.224924 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:14.226863 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.227161 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.227163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.227163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.227218 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.227422 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.227423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.227423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.227547 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:14.227614 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:14.227793 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 11:17:14.227886 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260927 11:17:14.229300 22818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9: Bootstrap starting.
I20260927 11:17:14.229847 22818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:14.229902 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.222628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.0.98.62:37523)
0927 11:17:14.222678 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.222680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.222687 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:17:14.222704 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.222705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.222713 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.222766 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.223329 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.223906 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.224700 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.224978 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.225023 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.227288 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:14.227289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:14.227290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:14.227296 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:14.228899 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.229225 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.229226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.229226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.229278 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.229449 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.229450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.229451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.229546 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:14.229704 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:14.229738 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 11:17:14.229812 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 11:17:14.230273 22818 log.cc:824] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:14.231016 22818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9: No bootstrap required, opened a new log
I20260927 11:17:14.232743 22818 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } }
I20260927 11:17:14.232929 22818 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce01f6045504425f857a6bc84021cfc9, State: Initialized, Role: FOLLOWER
I20260927 11:17:14.233039 22818 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } }
I20260927 11:17:14.233299 22825 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } } }
I20260927 11:17:14.233363 22818 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:14.233377 22825 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9 [sys.catalog]: This master's current role is: FOLLOWER
Sep 27 11:17:14 dist-test-slave-9hq9 krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260927 11:17:14.235316 22836 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:17:14.235435 22836 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:17:14.235463 22750 master_runner.cc:186] Error getting master registration for 127.0.98.62:37523: OK, instance_id { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" instance_seqno: 1790507833914255 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 11:17:14.237799 22750 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 11:17:14.238125 22747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.230590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57788 (local address 127.0.98.61:40659)
0927 11:17:14.230624 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.230625 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.230634 (+ 9us) server_negotiation.cc:407] Connection header received
0927 11:17:14.230682 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.230683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.230690 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.230760 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.231182 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.231873 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.232485 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.232786 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.232862 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.235203 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:14.235204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:14.235204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:14.235210 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:14.236902 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.237272 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.237273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.237273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.237327 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.237515 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.237515 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.237516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.237622 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:14.237736 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:14.237951 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 11:17:14.238018 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:17:14.239589 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:17:14.240444 22642 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e642c466bca941c49ae7bcc73036476c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34736
I20260927 11:17:14.253484 22668 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } attempt: 2
Sep 27 11:17:14 dist-test-slave-9hq9 krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:17:14.260799 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.253715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42017 (local address 127.0.0.1:42110)
0927 11:17:14.253760 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.253769 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:14.253802 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:14.253946 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:14.253947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:14.254118 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:14.254156 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.254157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.254728 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.254729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.255526 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.255528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.256115 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.256116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.256167 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.256600 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:14.256605 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:14.257864 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:14.259721 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.259722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.259722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.259947 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.260158 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.260159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.260159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.260218 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.260509 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:14.260510 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:14.260551 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 11:17:14.260618 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 11:17:14.260701 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 11:17:14.260829 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.253792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.0.98.60:42017)
0927 11:17:14.253848 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.253850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.253857 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:17:14.253873 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.253874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.253888 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.253942 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.254218 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.254644 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.255643 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.256011 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.256117 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.257944 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:14.257945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:14.257946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:14.257952 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:14.259640 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.260025 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.260026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.260027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.260083 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.260280 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.260281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.260281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.260442 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:14.260509 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:14.260667 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 11:17:14.260742 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 11:17:14.262393 22668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c: Bootstrap starting.
I20260927 11:17:14.263096 22668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:14.263410 22668 log.cc:824] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:14.264266 22668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c: No bootstrap required, opened a new log
I20260927 11:17:14.265637 22668 raft_consensus.cc:348] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } }
I20260927 11:17:14.265774 22668 raft_consensus.cc:729] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e642c466bca941c49ae7bcc73036476c, State: Initialized, Role: FOLLOWER
I20260927 11:17:14.265858 22668 consensus_queue.cc:260] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } }
I20260927 11:17:14.266151 22839 sys_catalog.cc:455] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } } }
I20260927 11:17:14.266234 22642 raft_consensus.cc:482] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 11:17:14.266278 22839 sys_catalog.cc:458] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:17:14.266192 22668 sys_catalog.cc:564] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:14.266342 22642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:14.267019 22642 raft_consensus.cc:504] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } }
I20260927 11:17:14.267258 22642 leader_election.cc:288] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [CANDIDATE]: Term 1 election: Requested vote from peers cff5b6c093f6461d89ca12664e0aa5f3 (127.0.98.61:40659), ce01f6045504425f857a6bc84021cfc9 (127.0.98.60:42017)
I20260927 11:17:14.267558 22792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e642c466bca941c49ae7bcc73036476c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ce01f6045504425f857a6bc84021cfc9"
I20260927 11:17:14.267688 22792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:14.268617 22792 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e642c466bca941c49ae7bcc73036476c in term 1.
I20260927 11:17:14.268842 22618 leader_election.cc:302] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce01f6045504425f857a6bc84021cfc9, e642c466bca941c49ae7bcc73036476c; no voters:
I20260927 11:17:14.269510 22839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:17:14.269642 22839 raft_consensus.cc:686] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [term 1 LEADER]: Becoming Leader. State: Replica: e642c466bca941c49ae7bcc73036476c, State: Running, Role: LEADER
I20260927 11:17:14.269727 22839 consensus_queue.cc:237] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } }
I20260927 11:17:14.270164 22839 sys_catalog.cc:455] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e642c466bca941c49ae7bcc73036476c. Latest consensus state: current_term: 1 leader_uuid: "e642c466bca941c49ae7bcc73036476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } } }
I20260927 11:17:14.270229 22839 sys_catalog.cc:458] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:14.271595 22600 master_runner.cc:186] Error getting master registration for 127.0.98.61:40659: OK, instance_id { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" instance_seqno: 1790507834054518 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 11:17:14.273916 22600 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 11:17:14.274086 22850 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:17:14.274209 22850 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:17:14.274319 22859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:17:14.274708 22859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:17:14.278168 22792 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9 [term 1 FOLLOWER]: Refusing update from remote peer e642c466bca941c49ae7bcc73036476c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:17:14.278582 22839 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:17:14.279779 22747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.270120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.0.98.61:40659)
0927 11:17:14.270160 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.270161 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.271549 (+ 1388us) server_negotiation.cc:407] Connection header received
0927 11:17:14.271652 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.271653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.271662 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.271691 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.272430 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.273112 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.274151 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.274901 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.274949 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.276739 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:14.276741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:14.276741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:14.276748 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:14.278479 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.278875 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.278876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.278876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.278947 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.279179 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.279180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.279180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.279299 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:14.279377 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:14.279598 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 11:17:14.279673 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:17:14.280457 22825 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e642c466bca941c49ae7bcc73036476c. Latest consensus state: current_term: 1 leader_uuid: "e642c466bca941c49ae7bcc73036476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } } }
I20260927 11:17:14.280568 22825 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:17:14.281929 22839 sys_catalog.cc:455] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e642c466bca941c49ae7bcc73036476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } } }
I20260927 11:17:14.282003 22839 sys_catalog.cc:458] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:14.282007 22859 catalog_manager.cc:1347] Generated new cluster ID: 722675fb2df44066a98974b0eed36cf4
I20260927 11:17:14.282084 22859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 11:17:14.286082 22743 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } attempt: 3
I20260927 11:17:14.286087 22825 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e642c466bca941c49ae7bcc73036476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } } }
I20260927 11:17:14.286167 22825 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:17:14.288606 22863 mvcc.cc:205] Tried to move back new op lower bound from 7333920089199259648 to 7333920089169088512. Current Snapshot: MvccSnapshot[applied={T|T < 7333920089199259648}]
I20260927 11:17:14.292297 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.270450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42116 (local address 127.0.98.60:42017)
0927 11:17:14.270481 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.270482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.274205 (+ 3723us) server_negotiation.cc:407] Connection header received
0927 11:17:14.274310 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.274311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.274320 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.274347 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.276007 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.276686 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.286249 (+ 9563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.286542 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.286581 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.288394 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:14.288395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:14.288395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:14.288401 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:14.290625 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.290986 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.290987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.290987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.291047 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.291251 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.291252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.291252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.291383 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:14.291778 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:14.292154 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 11:17:14.292217 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:17:14.292759 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.270901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34750 (local address 127.0.98.62:37523)
0927 11:17:14.270947 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.270949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.275156 (+ 4207us) server_negotiation.cc:407] Connection header received
0927 11:17:14.275248 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.275249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.275258 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.275287 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.277641 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.278385 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.286898 (+ 8513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.287185 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.287222 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.289275 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:14.289276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:14.289277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:14.289283 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:14.291324 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.291740 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.291742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.291742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.291794 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.291982 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.291983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.291983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.292112 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:14.292207 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:14.292552 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 11:17:14.292628 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 27 11:17:14 dist-test-slave-9hq9 krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:17:14.295667 22745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.286301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42017 (local address 127.0.0.1:42120)
0927 11:17:14.286351 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:14.286361 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:14.286402 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:14.286522 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:14.286523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:14.286662 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:14.286693 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.286694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.287540 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.287541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.288401 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:14.288404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.288945 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.288946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:14.289011 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.291615 (+ 2604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:14.291620 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:14.293081 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:14.294771 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.294772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.294772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.294976 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.295163 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:14.295164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:14.295164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:14.295210 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:14.295418 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:14.295418 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:14.295464 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 11:17:14.295514 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 11:17:14.295586 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 11:17:14.295718 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:14.286383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42120 (local address 127.0.98.60:42017)
0927 11:17:14.286415 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.286416 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.286423 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:17:14.286442 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.286442 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.286451 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.286511 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.286745 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.287448 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.288487 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:14.288829 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:14.288893 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:14.293162 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:14.293163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:14.293163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:14.293169 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:14.294694 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.295040 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.295041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.295041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.295094 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:14.295253 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:14.295253 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:14.295254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:14.295365 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:14.295421 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:14.295549 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 11:17:14.295619 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:17:14.297205 22743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3: Bootstrap starting.
I20260927 11:17:14.297901 22743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:14.298218 22743 log.cc:824] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:14.310660 22743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3: No bootstrap required, opened a new log
I20260927 11:17:14.312450 22743 raft_consensus.cc:348] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } }
I20260927 11:17:14.312561 22743 raft_consensus.cc:729] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff5b6c093f6461d89ca12664e0aa5f3, State: Initialized, Role: FOLLOWER
I20260927 11:17:14.312621 22743 consensus_queue.cc:260] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } }
I20260927 11:17:14.312870 22743 sys_catalog.cc:564] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:14.314194 22717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e642c466bca941c49ae7bcc73036476c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cff5b6c093f6461d89ca12664e0aa5f3"
I20260927 11:17:14.314298 22717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:14.315140 22717 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e642c466bca941c49ae7bcc73036476c in term 1.
I20260927 11:17:14.315485 22716 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3 [term 1 FOLLOWER]: Refusing update from remote peer e642c466bca941c49ae7bcc73036476c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:17:14.315833 22866 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:17:14.317939 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e642c466bca941c49ae7bcc73036476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } } }
I20260927 11:17:14.318199 22852 sys_catalog.cc:455] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e642c466bca941c49ae7bcc73036476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } } }
I20260927 11:17:14.318284 22852 sys_catalog.cc:458] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:14.318089 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:17:14.318642 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e642c466bca941c49ae7bcc73036476c. Latest consensus state: current_term: 1 leader_uuid: "e642c466bca941c49ae7bcc73036476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } } }
I20260927 11:17:14.318701 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:17:14.318729 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e642c466bca941c49ae7bcc73036476c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e642c466bca941c49ae7bcc73036476c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37523 } } peers { permanent_uuid: "cff5b6c093f6461d89ca12664e0aa5f3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40659 } } peers { permanent_uuid: "ce01f6045504425f857a6bc84021cfc9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42017 } } }
I20260927 11:17:14.318771 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:17:14.319206 22859 catalog_manager.cc:1370] Generated new certificate authority record
W20260927 11:17:14.319720 22880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:17:14.319774 22880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:17:14.320015 22859 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:17:14.320151 22882 mvcc.cc:205] Tried to move back new op lower bound from 7333920089199259648 to 7333920089169088512. Current Snapshot: MvccSnapshot[applied={T|T < 7333920089199259648}]
I20260927 11:17:14.325359 22859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e642c466bca941c49ae7bcc73036476c: Generated new TSK 0
I20260927 11:17:14.325559 22859 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 11:17:14.349217 22671 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:14.347763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34754 (local address 127.0.98.62:37523)
0927 11:17:14.347800 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.347802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.348608 (+ 806us) server_negotiation.cc:407] Connection header received
0927 11:17:14.348697 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.348698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.348725 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.348755 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.349159 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34754: BlockingRecv error: recv got EOF from 127.0.0.1:34754 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 11:17:14.350100 22747 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:14.348073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.0.98.61:40659)
0927 11:17:14.348125 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.348127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.349421 (+ 1294us) server_negotiation.cc:407] Connection header received
0927 11:17:14.349677 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.349678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.349688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.349715 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.350022 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57814: BlockingRecv error: recv got EOF from 127.0.0.1:57814 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 11:17:14.350612 22822 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:14.348347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42126 (local address 127.0.98.60:42017)
0927 11:17:14.348385 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.348387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.349543 (+ 1156us) server_negotiation.cc:407] Connection header received
0927 11:17:14.350238 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.350239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.350249 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.350277 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.350560 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42126: BlockingRecv error: recv got EOF from 127.0.0.1:42126 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 11:17:14.354864 22747 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:14.354273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.0.98.61:40659)
0927 11:17:14.354310 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.354311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.354413 (+ 102us) server_negotiation.cc:407] Connection header received
0927 11:17:14.354509 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.354509 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.354523 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.354550 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.354827 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57818: BlockingRecv error: recv got EOF from 127.0.0.1:57818 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 11:17:14.360848 22747 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:14.360114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57828 (local address 127.0.98.61:40659)
0927 11:17:14.360144 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.360145 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.360402 (+ 257us) server_negotiation.cc:407] Connection header received
0927 11:17:14.360491 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.360492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.360501 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.360528 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.360812 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57828: BlockingRecv error: recv got EOF from 127.0.0.1:57828 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 11:17:14.364717 22747 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:14.363831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.0.98.61:40659)
0927 11:17:14.363870 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.363872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.364222 (+ 350us) server_negotiation.cc:407] Connection header received
0927 11:17:14.364363 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.364364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.364373 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.364400 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.364679 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57844: BlockingRecv error: recv got EOF from 127.0.0.1:57844 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 11:17:14.370397 22747 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:14.369664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57852 (local address 127.0.98.61:40659)
0927 11:17:14.369693 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.369694 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.369896 (+ 202us) server_negotiation.cc:407] Connection header received
0927 11:17:14.369982 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.369983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.369992 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.370018 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.370360 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57852: BlockingRecv error: recv got EOF from 127.0.0.1:57852 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 11:17:14.380015 22747 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:14.379309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57866 (local address 127.0.98.61:40659)
0927 11:17:14.379339 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.379341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.379568 (+ 227us) server_negotiation.cc:407] Connection header received
0927 11:17:14.379655 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.379656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.379665 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.379692 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.379967 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57866: BlockingRecv error: recv got EOF from 127.0.0.1:57866 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260927 11:17:14.397912 22747 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:14.397058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.0.98.61:40659)
0927 11:17:14.397094 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.397095 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.397368 (+ 273us) server_negotiation.cc:407] Connection header received
0927 11:17:14.397483 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.397484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.397494 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.397523 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.397867 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57874: BlockingRecv error: recv got EOF from 127.0.0.1:57874 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 11:17:14.432004 22747 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:14.431245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.0.98.61:40659)
0927 11:17:14.431293 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.431295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.431407 (+ 112us) server_negotiation.cc:407] Connection header received
0927 11:17:14.431468 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.431469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.431480 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.431560 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.431947 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42326: BlockingRecv error: recv got EOF from 127.0.0.1:42326 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 11:17:14.497882 22747 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:14.497207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.0.98.61:40659)
0927 11:17:14.497280 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.497282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.497321 (+ 39us) server_negotiation.cc:407] Connection header received
0927 11:17:14.497400 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.497401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.497411 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.497463 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.497785 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42330: BlockingRecv error: recv got EOF from 127.0.0.1:42330 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 11:17:14.627573 22747 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:14.626886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42342 (local address 127.0.98.61:40659)
0927 11:17:14.626948 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.626950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.627046 (+ 96us) server_negotiation.cc:407] Connection header received
0927 11:17:14.627115 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.627116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.627126 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.627170 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.627495 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42342: BlockingRecv error: recv got EOF from 127.0.0.1:42342 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 11:17:14.885423 22747 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:14.884654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.0.98.61:40659)
0927 11:17:14.884719 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:14.884722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:14.884794 (+ 72us) server_negotiation.cc:407] Connection header received
0927 11:17:14.884889 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:14.884890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:14.884901 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:14.884980 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:14.885360 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42356: BlockingRecv error: recv got EOF from 127.0.0.1:42356 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 11:17:15.235893 22836 catalog_manager.cc:1259] Loaded cluster ID: 722675fb2df44066a98974b0eed36cf4
I20260927 11:17:15.235950 22836 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9: loading cluster ID for follower catalog manager: success
I20260927 11:17:15.236651 22836 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9: acquiring CA information for follower catalog manager: success
I20260927 11:17:15.236937 22836 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ce01f6045504425f857a6bc84021cfc9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 11:17:15.320241 22880 catalog_manager.cc:1259] Loaded cluster ID: 722675fb2df44066a98974b0eed36cf4
I20260927 11:17:15.320334 22880 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3: loading cluster ID for follower catalog manager: success
I20260927 11:17:15.321128 22880 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3: acquiring CA information for follower catalog manager: success
I20260927 11:17:15.321435 22880 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cff5b6c093f6461d89ca12664e0aa5f3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 11:17:15.400815 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:15.398543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42366 (local address 127.0.98.61:40659)
0927 11:17:15.398667 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:15.398669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:15.398679 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:17:15.398730 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:15.398731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:15.398775 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:15.398845 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:17:15.398980 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:15.399463 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:15.400172 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:15.400361 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:15.400412 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:15.400544 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:17:15.400645 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:17:15.400700 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:15.400739 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 11:17:15.400755 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260927 11:17:15.404624 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:15.402555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41718 (local address 127.0.98.60:42017)
0927 11:17:15.402663 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:15.402664 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:15.402675 (+ 11us) server_negotiation.cc:407] Connection header received
0927 11:17:15.402747 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:15.402748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:15.402780 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:15.402842 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:17:15.402919 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:15.403379 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:15.404029 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:15.404216 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:15.404262 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:15.404382 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:17:15.404465 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:17:15.404517 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:15.404553 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 11:17:15.404568 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":69,"threads_started":1}
I20260927 11:17:15.408846 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:15.406136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42378 (local address 127.0.98.61:40659)
0927 11:17:15.406173 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:15.406175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:15.406286 (+ 111us) server_negotiation.cc:407] Connection header received
0927 11:17:15.406414 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:15.406415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:15.406425 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:15.406482 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:17:15.406609 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:15.407309 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:15.408132 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:15.408406 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:15.408469 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:15.408585 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:17:15.408615 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:17:15.408665 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:15.408755 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 11:17:15.408775 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 11:17:15.409073 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:15.406383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.0.98.60:42017)
0927 11:17:15.406429 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:15.406430 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:15.406526 (+ 96us) server_negotiation.cc:407] Connection header received
0927 11:17:15.406612 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:15.406613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:15.406621 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:15.406649 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:17:15.406815 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:15.407244 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:15.408267 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:15.408563 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:15.408621 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:15.408830 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:17:15.408857 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:17:15.408885 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:15.409001 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 11:17:15.409020 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 11:17:15.409389 22966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:15.406134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48346 (local address 127.0.98.62:37523)
0927 11:17:15.406361 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:15.406364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:15.406372 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:17:15.406412 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:15.406413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:15.406457 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:15.406565 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:17:15.406699 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:15.407434 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:15.408385 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:15.408702 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:15.408955 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:15.409025 (+ 70us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:17:15.409117 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:17:15.409219 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:15.409278 (+ 59us) server_negotiation.cc:299] Negotiation successful
0927 11:17:15.409301 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":56,"threads_started":1}
I20260927 11:17:15.411127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 22600
I20260927 11:17:15.416442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 22675
I20260927 11:17:15.421433 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 22750
2026-09-27T11:17:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2750 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 11:17:15 dist-test-slave-9hq9 krb5kdc[22970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 11:17:15 dist-test-slave-9hq9 krb5kdc[22970](info): set up 2 sockets
Sep 27 11:17:15 dist-test-slave-9hq9 krb5kdc[22970](info): commencing operation
krb5kdc: starting...
W20260927 11:17:16.442267 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/test-user.keytab.
Sep 27 11:17:16 dist-test-slave-9hq9 krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T11:17:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T11:17:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 11:17:16.550838 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41131
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41131,127.0.98.61:42539,127.0.98.60:38241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 11:17:16.627164 22983 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 11:17:16.627336 22983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:16.627351 22983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:16.627389 22983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:16.628397 22983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:16.628428 22983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:16.628443 22983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:16.628455 22983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:16.628468 22983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:16.629786 22983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41131,127.0.98.61:42539,127.0.98.60:38241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:16.629948 22983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:16.630111 22983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:17:16.634501 22991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:16.634475 22989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:16.634552 22988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:16.634708 22983 server_base.cc:1034] running on GCE node
I20260927 11:17:16.634935 22983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:16.635241 22983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:16.636400 22983 hybrid_clock.cc:648] HybridClock initialized: now 1790507836636391 us; error 38 us; skew 500 ppm
Sep 27 11:17:16 dist-test-slave-9hq9 krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:16.638721 22983 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 11:17:16.639106 22983 webserver.cc:460] Webserver started at http://127.0.98.62:32965/ using document root <none> and password file <none>
I20260927 11:17:16.639320 22983 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:16.639384 22983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:16.639524 22983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:16.640547 22983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/data/instance:
uuid: "35d90a5f4d4047f8b7817f00d9da5a5e"
format_stamp: "Formatted at 2026-09-27 11:17:16 on dist-test-slave-9hq9"
server_key: "4011adc0b47fe933906844b66e4dc0e4"
server_key_iv: "b74b03d4780f04553249c725c82fc257"
server_key_version: "encryptionkey@0"
I20260927 11:17:16.640875 22983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/wal/instance:
uuid: "35d90a5f4d4047f8b7817f00d9da5a5e"
format_stamp: "Formatted at 2026-09-27 11:17:16 on dist-test-slave-9hq9"
server_key: "4011adc0b47fe933906844b66e4dc0e4"
server_key_iv: "b74b03d4780f04553249c725c82fc257"
server_key_version: "encryptionkey@0"
I20260927 11:17:16.642185 22983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 11:17:16.642861 22998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:16.643006 22983 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:17:16.643112 22983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/wal
uuid: "35d90a5f4d4047f8b7817f00d9da5a5e"
format_stamp: "Formatted at 2026-09-27 11:17:16 on dist-test-slave-9hq9"
server_key: "4011adc0b47fe933906844b66e4dc0e4"
server_key_iv: "b74b03d4780f04553249c725c82fc257"
server_key_version: "encryptionkey@0"
I20260927 11:17:16.643188 22983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:16.667439 22983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:16.669165 22983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:16.669325 22983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:16.673126 22983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41131
I20260927 11:17:16.673159 23050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41131 every 8 connection(s)
I20260927 11:17:16.673470 22983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/data/info.pb
I20260927 11:17:16.674456 23051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:16.675017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 22983
I20260927 11:17:16.675104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-0/wal/instance
I20260927 11:17:16.675257 392 external_mini_cluster.cc:1428] Setting key 6a3b87ea9e55c319ba426e9c4467eace
I20260927 11:17:16.675849 23051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:17:16 dist-test-slave-9hq9 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 11:17:16.685685 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.676467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41131 (local address 127.0.0.1:33902)
0927 11:17:16.676620 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.676656 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:16.677078 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:16.677356 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:16.677358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:16.677565 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:16.677705 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:16.677709 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.679286 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.679287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:16.679916 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:16.679917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.680642 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.680643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:16.680693 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.681193 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:16.681200 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:16.682511 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:16.684176 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:16.684182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:16.684186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:16.684418 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:16.684661 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:16.684662 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:16.684662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:16.684778 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:16.685186 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:16.685191 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:16.685247 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 11:17:16.685331 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 11:17:16.685541 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":91,"thread_start_us":88,"threads_started":1}
I20260927 11:17:16.685688 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.676656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33902 (local address 127.0.98.62:41131)
0927 11:17:16.676803 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:16.676808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:16.676847 (+ 39us) server_negotiation.cc:407] Connection header received
0927 11:17:16.677224 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:16.677226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:16.677248 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:16.677351 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:16.677793 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.679192 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.680000 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.680537 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.680652 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.682597 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:16.682605 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:16.682606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:16.682613 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:16.684069 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.684500 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.684503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.684504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.684585 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.684844 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.684845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.684845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.685088 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:16.685184 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:16.685368 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 11:17:16.685474 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260927 11:17:16.686395 23051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:17:16.686715 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.686595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42539 (local address 127.0.0.1:33096)
0927 11:17:16.686648 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.686692 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 11:17:16.686874 23000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42539: connect: Connection refused (error 111)
I20260927 11:17:16.687292 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.687194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42539 (local address 127.0.0.1:33112)
0927 11:17:16.687245 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.687271 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 11:17:16.687399 23051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42539: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42539: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 11:17:16.698601 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42539
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41131,127.0.98.61:42539,127.0.98.60:38241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:17:16.736611 23051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } attempt: 1
I20260927 11:17:16.737066 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.736943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42539 (local address 127.0.0.1:33116)
0927 11:17:16.736999 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.737035 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 11:17:16.737418 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.737331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42539 (local address 127.0.0.1:33122)
0927 11:17:16.737374 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.737399 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 11:17:16.737509 23051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42539: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42539: connect: Connection refused (error 111)
W20260927 11:17:16.775645 23058 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 11:17:16.775818 23058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:16.775835 23058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:16.775853 23058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:16.776889 23058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:16.776922 23058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:16.776935 23058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:16.776948 23058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:16.776960 23058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:16.778317 23058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41131,127.0.98.61:42539,127.0.98.60:38241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:16.778477 23058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:16.778647 23058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 11:17:16.783001 23063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:16.782979 23066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:16.782986 23064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:16.783077 23058 server_base.cc:1034] running on GCE node
I20260927 11:17:16.783366 23058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:16.783633 23058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:16.784796 23058 hybrid_clock.cc:648] HybridClock initialized: now 1790507836784783 us; error 31 us; skew 500 ppm
Sep 27 11:17:16 dist-test-slave-9hq9 krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:16.787089 23058 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 11:17:16.787424 23058 webserver.cc:460] Webserver started at http://127.0.98.61:46873/ using document root <none> and password file <none>
I20260927 11:17:16.787577 23058 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:16.787613 23058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:16.787691 23058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:16.788558 23058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/data/instance:
uuid: "66ce752816af4f8da7fa4f958b9777a3"
format_stamp: "Formatted at 2026-09-27 11:17:16 on dist-test-slave-9hq9"
server_key: "4279f764df899099a72874afdfc94ad3"
server_key_iv: "4fcd07fdf35e7a2319e5e2af0e83e1d4"
server_key_version: "encryptionkey@0"
I20260927 11:17:16.788805 23058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/wal/instance:
uuid: "66ce752816af4f8da7fa4f958b9777a3"
format_stamp: "Formatted at 2026-09-27 11:17:16 on dist-test-slave-9hq9"
server_key: "4279f764df899099a72874afdfc94ad3"
server_key_iv: "4fcd07fdf35e7a2319e5e2af0e83e1d4"
server_key_version: "encryptionkey@0"
I20260927 11:17:16.789927 23058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 11:17:16.790669 23073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:16.790833 23058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:16.790911 23058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/wal
uuid: "66ce752816af4f8da7fa4f958b9777a3"
format_stamp: "Formatted at 2026-09-27 11:17:16 on dist-test-slave-9hq9"
server_key: "4279f764df899099a72874afdfc94ad3"
server_key_iv: "4fcd07fdf35e7a2319e5e2af0e83e1d4"
server_key_version: "encryptionkey@0"
I20260927 11:17:16.790962 23058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:16.805740 23051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } attempt: 2
I20260927 11:17:16.806133 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.805989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42539 (local address 127.0.0.1:33136)
0927 11:17:16.806046 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.806092 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 11:17:16.806502 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.806426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42539 (local address 127.0.0.1:33140)
0927 11:17:16.806464 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.806487 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 11:17:16.806595 23051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42539: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42539: connect: Connection refused (error 111)
I20260927 11:17:16.813489 23058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:16.815248 23058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:16.815344 23058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:16.818878 23058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42539
I20260927 11:17:16.818949 23125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42539 every 8 connection(s)
I20260927 11:17:16.819248 23058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/data/info.pb
I20260927 11:17:16.820082 23126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:16.821434 23126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:17:16.823065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 23058
I20260927 11:17:16.823170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-1/wal/instance
I20260927 11:17:16.823350 392 external_mini_cluster.cc:1428] Setting key 6853dd4ef5a3bab38d025e85f5e360f9
Sep 27 11:17:16 dist-test-slave-9hq9 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:17:16.830513 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.821990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41131 (local address 127.0.0.1:33918)
0927 11:17:16.822178 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.822209 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:16.822656 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:16.822855 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:16.822857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:16.823067 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:16.823313 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:16.823320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.824234 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.824235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:16.824975 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:16.824977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.825394 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.825396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:16.825520 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.825909 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:16.825918 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:16.827461 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:16.829258 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:16.829264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:16.829265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:16.829465 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:16.829671 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:16.829672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:16.829672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:16.829762 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:16.830039 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:16.830042 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:16.830188 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 11:17:16.830274 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 11:17:16.830378 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":115,"threads_started":1}
I20260927 11:17:16.830510 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.822025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.0.98.62:41131)
0927 11:17:16.822115 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:16.822118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:16.822277 (+ 159us) server_negotiation.cc:407] Connection header received
0927 11:17:16.822736 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:16.822737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:16.822748 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:16.822798 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:16.823408 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.824143 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.825078 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.825287 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.825370 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.827549 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:16.827551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:16.827551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:16.827558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:16.829175 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.829540 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.829541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.829542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.829601 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.829826 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.829827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.829827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.829970 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:16.830045 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:16.830324 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 11:17:16.830399 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 11:17:16.831046 23126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 11:17:16 dist-test-slave-9hq9 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:17:16.838161 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.831281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42539 (local address 127.0.0.1:33142)
0927 11:17:16.831334 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.831344 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:16.831389 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:16.831612 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:16.831613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:16.831773 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:16.831827 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:16.831828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.832590 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.832591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:16.833099 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:16.833100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.833505 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.833505 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:16.833554 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.833850 (+ 296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:16.833854 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:16.835138 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:16.837141 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:16.837142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:16.837142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:16.837302 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:16.837510 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:16.837511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:16.837512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:16.837564 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:16.837840 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:16.837841 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:16.837876 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 11:17:16.837959 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 11:17:16.838045 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 11:17:16.838233 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.831332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33142 (local address 127.0.98.61:42539)
0927 11:17:16.831466 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:16.831470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:16.831487 (+ 17us) server_negotiation.cc:407] Connection header received
0927 11:17:16.831510 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:16.831516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:16.831527 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:16.831586 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:16.831893 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.832520 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.833162 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.833440 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.833494 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.835237 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:16.835244 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:16.835244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:16.835250 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:16.837031 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.837388 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.837390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.837390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.837440 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.837637 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.837637 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.837638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.837772 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:16.837858 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:16.837987 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 11:17:16.838112 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
I20260927 11:17:16.838716 23126 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:17:16.839041 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.838908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38241 (local address 127.0.0.1:37210)
0927 11:17:16.838958 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.839011 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 11:17:16.839174 23075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
I20260927 11:17:16.839591 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.839479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38241 (local address 127.0.0.1:37222)
0927 11:17:16.839531 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.839561 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 11:17:16.839700 23126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38241: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 11:17:16.849113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskGpH_nd/build/release/bin/kudu
/tmp/dist-test-taskGpH_nd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38241
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41131,127.0.98.61:42539,127.0.98.60:38241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 11:17:16.888998 23126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } attempt: 1
I20260927 11:17:16.889405 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.889287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38241 (local address 127.0.0.1:37226)
0927 11:17:16.889347 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.889381 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 11:17:16.889739 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.889641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38241 (local address 127.0.0.1:37232)
0927 11:17:16.889692 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.889715 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 11:17:16.889815 23126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38241: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
I20260927 11:17:16.897751 23051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } attempt: 3
Sep 27 11:17:16 dist-test-slave-9hq9 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:17:16.904688 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.898207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33150 (local address 127.0.98.61:42539)
0927 11:17:16.898263 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:16.898265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:16.898303 (+ 38us) server_negotiation.cc:407] Connection header received
0927 11:17:16.898381 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:16.898382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:16.898390 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:16.898436 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:16.898754 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.899217 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.900195 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.900388 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.900441 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.902294 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:16.902295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:16.902296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:16.902303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:16.903573 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.903919 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.903920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.903920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.903973 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.904192 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.904193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.904194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.904298 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:16.904367 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:16.904522 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 11:17:16.904596 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 11:17:16.904706 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.898193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42539 (local address 127.0.0.1:33150)
0927 11:17:16.898247 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.898257 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:16.898321 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:16.898442 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:16.898443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:16.898608 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:16.898680 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:16.898681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.899303 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.899304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:16.900117 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:16.900120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.900453 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.900454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:16.900529 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.900915 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:16.900920 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:16.902215 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:16.903664 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:16.903664 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:16.903665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:16.903838 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:16.904047 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:16.904047 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:16.904048 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:16.904127 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:16.904371 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:16.904372 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:16.904416 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 11:17:16.904499 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 11:17:16.904595 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 11:17:16.905055 23051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:17:16.905380 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.905285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38241 (local address 127.0.0.1:37236)
0927 11:17:16.905340 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.905364 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 11:17:16.905668 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.905559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38241 (local address 127.0.0.1:37238)
0927 11:17:16.905620 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.905642 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 11:17:16.905757 23051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38241: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
W20260927 11:17:16.928527 23133 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 11:17:16.928714 23133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 11:17:16.928730 23133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 11:17:16.928750 23133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 11:17:16.929839 23133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 11:17:16.929889 23133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 11:17:16.929903 23133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 11:17:16.929934 23133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 11:17:16.929958 23133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 11:17:16.931349 23133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41131,127.0.98.61:42539,127.0.98.60:38241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 10:19:16 UTC on ccycloud.quasar-rhysdl.root.comops.site
build id 4675881
I20260927 11:17:16.931540 23133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 11:17:16.931733 23133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 11:17:16.936915 23051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } attempt: 1
W20260927 11:17:16.937115 23138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:16.937208 23141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 11:17:16.937295 23139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 11:17:16.937336 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.937211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38241 (local address 127.0.0.1:37248)
0927 11:17:16.937273 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.937299 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 11:17:16.937621 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.937530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38241 (local address 127.0.0.1:37258)
0927 11:17:16.937576 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.937600 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 11:17:16.937685 23133 server_base.cc:1034] running on GCE node
W20260927 11:17:16.937753 23051 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38241: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
I20260927 11:17:16.937949 23133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 11:17:16.938249 23133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 11:17:16.939405 23133 hybrid_clock.cc:648] HybridClock initialized: now 1790507836939396 us; error 25 us; skew 500 ppm
Sep 27 11:17:16 dist-test-slave-9hq9 krb5kdc[22970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 11:17:16.941792 23133 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 11:17:16.942214 23133 webserver.cc:460] Webserver started at http://127.0.98.60:33073/ using document root <none> and password file <none>
I20260927 11:17:16.942384 23133 fs_manager.cc:359] Metadata directory not provided
I20260927 11:17:16.942422 23133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 11:17:16.942507 23133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 11:17:16.943421 23133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/data/instance:
uuid: "3ee212dd44434c5cb56988f7f2678bdf"
format_stamp: "Formatted at 2026-09-27 11:17:16 on dist-test-slave-9hq9"
server_key: "05a57646d404f56d171170e40dec0761"
server_key_iv: "a53d784e3ee51ce9eb7e34be6f48f7a7"
server_key_version: "encryptionkey@0"
I20260927 11:17:16.943707 23133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/wal/instance:
uuid: "3ee212dd44434c5cb56988f7f2678bdf"
format_stamp: "Formatted at 2026-09-27 11:17:16 on dist-test-slave-9hq9"
server_key: "05a57646d404f56d171170e40dec0761"
server_key_iv: "a53d784e3ee51ce9eb7e34be6f48f7a7"
server_key_version: "encryptionkey@0"
I20260927 11:17:16.944880 23133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 11:17:16.945482 23148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 11:17:16.945637 23133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 11:17:16.945712 23133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/wal
uuid: "3ee212dd44434c5cb56988f7f2678bdf"
format_stamp: "Formatted at 2026-09-27 11:17:16 on dist-test-slave-9hq9"
server_key: "05a57646d404f56d171170e40dec0761"
server_key_iv: "a53d784e3ee51ce9eb7e34be6f48f7a7"
server_key_version: "encryptionkey@0"
I20260927 11:17:16.945780 23133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 11:17:16.956316 23133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 11:17:16.957957 23126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } attempt: 2
I20260927 11:17:16.958364 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.958252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38241 (local address 127.0.0.1:37268)
0927 11:17:16.958308 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.958340 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 11:17:16.958496 23133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 11:17:16.958655 23133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 11:17:16.958768 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.958691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38241 (local address 127.0.0.1:37270)
0927 11:17:16.958731 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.958753 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 11:17:16.958860 23126 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38241: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38241: connect: Connection refused (error 111)
I20260927 11:17:16.962355 23133 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38241
I20260927 11:17:16.962381 23200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38241 every 8 connection(s)
I20260927 11:17:16.962704 23133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/data/info.pb
I20260927 11:17:16.963352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskGpH_nd/build/release/bin/kudu as pid 23133
I20260927 11:17:16.963447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskGpH_nd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790507576023819-392-0/minicluster-data/master-2/wal/instance
I20260927 11:17:16.963631 392 external_mini_cluster.cc:1428] Setting key 2f8f5c6cfe2edf473d3b5ace27c62d4b
I20260927 11:17:16.963842 23201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 11:17:16.966076 23201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 11:17:16 dist-test-slave-9hq9 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 11:17:16 dist-test-slave-9hq9 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:17:16.974936 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.966728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.0.98.62:41131)
0927 11:17:16.966767 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:16.966769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:16.966926 (+ 157us) server_negotiation.cc:407] Connection header received
0927 11:17:16.966984 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:16.966984 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:16.966993 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:16.967037 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:16.967601 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.968242 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.969293 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.969702 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.969769 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.971935 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:16.971936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:16.971937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:16.971943 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:16.973531 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.973894 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.973896 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.973896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.973954 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.974242 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.974243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.974243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.974373 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:16.974422 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:16.974755 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 11:17:16.974848 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:17:16.975045 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.966641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41131 (local address 127.0.0.1:33934)
0927 11:17:16.966826 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.966861 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:16.966926 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:16.967053 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:16.967056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:16.967420 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:16.967502 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:16.967504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.968327 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.968328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:16.969180 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:16.969183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.969827 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.969828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:16.969942 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.970411 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:16.970420 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:16.971856 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:16.973607 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:16.973613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:16.973615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:16.973821 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:16.974025 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:16.974025 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:16.974026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:16.974164 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:16.974442 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:16.974448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:16.974595 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 11:17:16.974725 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 11:17:16.974877 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
I20260927 11:17:16.975553 23201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:17:16.975688 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.964116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37278 (local address 127.0.98.60:38241)
0927 11:17:16.964294 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:16.964302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:16.964333 (+ 31us) server_negotiation.cc:407] Connection header received
0927 11:17:16.964937 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:16.964946 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:16.965004 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:16.965150 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:16.965704 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.967043 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.968104 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.968703 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.968833 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.972566 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:16.972574 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:16.972574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:16.972581 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:16.974230 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.974562 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.974564 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.974564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.974608 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.974793 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.974793 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.974794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.974940 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:16.975071 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:16.975500 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 11:17:16.975566 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
Sep 27 11:17:16 dist-test-slave-9hq9 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 11:17:16 dist-test-slave-9hq9 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 11:17:16.982399 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.975793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42539 (local address 127.0.0.1:33152)
0927 11:17:16.975915 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.975924 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:16.975963 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:16.976325 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:16.976325 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:16.976489 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:16.976535 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:16.976537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.977247 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.977248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:16.977799 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:16.977801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.978187 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.978188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:16.978252 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.978625 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:16.978629 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:16.979832 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:16.981390 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:16.981391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:16.981391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:16.981550 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:16.981785 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:16.981786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:16.981786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:16.981845 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:16.982178 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:16.982179 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:16.982221 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 11:17:16.982278 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 11:17:16.982349 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 11:17:16.982517 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.975879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.0.98.61:42539)
0927 11:17:16.975959 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:16.975961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:16.975969 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:17:16.976201 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:16.976202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:16.976213 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:16.976279 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:16.976610 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.977166 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.977878 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.978116 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.978177 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.979915 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:16.979916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:16.979916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:16.979923 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:16.981290 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.981634 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.981635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.981636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.981699 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.981919 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.981920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.981920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.982023 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:16.982171 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:16.982334 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 11:17:16.982410 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 11:17:16.982851 23201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } has no permanent_uuid. Determining permanent_uuid...
I20260927 11:17:16.983188 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.976505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33940 (local address 127.0.98.62:41131)
0927 11:17:16.976535 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:16.976537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:16.976544 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:17:16.976607 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:16.976608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:16.976616 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:16.976663 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:16.977067 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.977546 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.978424 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.978691 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.978738 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.980681 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:16.980683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:16.980683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:16.980689 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:16.982004 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.982451 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.982452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.982453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.982510 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.982732 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.982733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.982733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.982857 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:16.982996 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:16.983034 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 11:17:16.983097 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
Sep 27 11:17:16 dist-test-slave-9hq9 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 11:17:16 dist-test-slave-9hq9 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 11:17:16.989513 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.983092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.0.98.60:38241)
0927 11:17:16.983121 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:16.983123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:16.983233 (+ 110us) server_negotiation.cc:407] Connection header received
0927 11:17:16.983252 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:16.983252 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:16.983261 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:16.983304 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:16.983572 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.984211 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.984900 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.985088 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.985135 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.986865 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:16.986866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:16.986866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:16.986872 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:16.988335 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.988697 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.988698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.988698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.988774 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.988988 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.988989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.988989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.989089 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:16.989151 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:16.989304 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 11:17:16.989406 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":13}
I20260927 11:17:16.989521 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.983067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38241 (local address 127.0.0.1:37292)
0927 11:17:16.983117 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:16.983127 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:16.983174 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:16.983331 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:16.983331 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:16.983456 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:16.983495 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:16.983497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.984286 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.984287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:16.984805 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:16.984807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.985154 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.985155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:16.985205 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.985521 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:16.985525 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:16.986780 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:16.988421 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:16.988421 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:16.988422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:16.988621 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:16.988843 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:16.988843 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:16.988844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:16.988920 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:16.989167 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:16.989168 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:16.989209 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 11:17:16.989280 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 11:17:16.989361 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 11:17:16.990723 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:16.983773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.0.98.61:42539)
0927 11:17:16.983831 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:16.983833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:16.983839 (+ 6us) server_negotiation.cc:407] Connection header received
0927 11:17:16.983855 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:16.983856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:16.983863 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:16.983915 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:16.984398 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.984802 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.985608 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:16.985784 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:16.985833 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:16.988049 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:16.988050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:16.988051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:16.988058 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:16.989655 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.989981 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.989982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.989983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.990031 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:16.990268 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:16.990269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:16.990269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:16.990374 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:16.990531 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:16.990566 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 11:17:16.990633 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 11:17:16.990890 23201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf: Bootstrap starting.
I20260927 11:17:16.991657 23201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:16.991935 23201 log.cc:824] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:16.992741 23201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf: No bootstrap required, opened a new log
I20260927 11:17:16.993484 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 11:17:16.994798 23201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } }
I20260927 11:17:16.994935 23201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee212dd44434c5cb56988f7f2678bdf, State: Initialized, Role: FOLLOWER
I20260927 11:17:16.995023 23201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } }
I20260927 11:17:16.995340 23201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:16.995319 23207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } } }
I20260927 11:17:16.995415 23207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:17:16.994668 23025 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33940
W20260927 11:17:16.997077 23218 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:17:16.997160 23218 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:17:16.997352 23133 master_runner.cc:186] Error getting master registration for 127.0.98.62:41131: OK, instance_id { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" instance_seqno: 1790507836672031 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 11:17:17.000039 23133 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 11:17:17.012916 23051 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } attempt: 2
Sep 27 11:17:17 dist-test-slave-9hq9 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:17:17.020231 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:17.013146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38241 (local address 127.0.0.1:37308)
0927 11:17:17.013190 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:17.013200 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:17.013234 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:17.013424 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:17.013425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:17.013590 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:17.013632 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:17.013634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:17.014513 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:17.014514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:17.015316 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:17.015318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:17.015757 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:17.015758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:17.015817 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:17.016190 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:17.016196 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:17.017601 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:17.019177 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:17.019178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:17.019178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:17.019405 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:17.019605 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:17.019606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:17.019606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:17.019668 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:17.019943 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:17.019944 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:17.019986 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 11:17:17.020058 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 11:17:17.020140 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 11:17:17.020318 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:17.013215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.0.98.60:38241)
0927 11:17:17.013254 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.013255 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.013263 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:17:17.013320 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.013321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.013329 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.013409 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.013709 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:17.014439 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:17.015397 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:17.015684 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:17.015749 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:17.017681 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:17.017683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:17.017683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:17.017689 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:17.019121 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:17.019484 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:17.019485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:17.019485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:17.019542 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:17.019731 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:17.019732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:17.019733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:17.019875 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:17.019937 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:17.020141 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 11:17:17.020224 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 11:17:17.021690 23051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e: Bootstrap starting.
I20260927 11:17:17.022393 23051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:17.022686 23051 log.cc:824] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:17.023479 23051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e: No bootstrap required, opened a new log
I20260927 11:17:17.024865 23051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } }
I20260927 11:17:17.024994 23051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d90a5f4d4047f8b7817f00d9da5a5e, State: Initialized, Role: FOLLOWER
I20260927 11:17:17.025068 23051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } }
I20260927 11:17:17.025363 23051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:17.025417 23025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 11:17:17.025481 23025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:17.025349 23221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } } }
I20260927 11:17:17.025667 23221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:17:17.026240 23025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } }
I20260927 11:17:17.026461 23025 leader_election.cc:288] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [CANDIDATE]: Term 1 election: Requested vote from peers 66ce752816af4f8da7fa4f958b9777a3 (127.0.98.61:42539), 3ee212dd44434c5cb56988f7f2678bdf (127.0.98.60:38241)
I20260927 11:17:17.026940 23175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3ee212dd44434c5cb56988f7f2678bdf"
I20260927 11:17:17.027064 23175 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:17.027913 23175 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35d90a5f4d4047f8b7817f00d9da5a5e in term 1.
I20260927 11:17:17.028136 23000 leader_election.cc:302] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35d90a5f4d4047f8b7817f00d9da5a5e, 3ee212dd44434c5cb56988f7f2678bdf; no voters:
I20260927 11:17:17.028337 23221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 11:17:17.028478 23221 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [term 1 LEADER]: Becoming Leader. State: Replica: 35d90a5f4d4047f8b7817f00d9da5a5e, State: Running, Role: LEADER
I20260927 11:17:17.028558 23221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } }
I20260927 11:17:17.028931 23221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35d90a5f4d4047f8b7817f00d9da5a5e. Latest consensus state: current_term: 1 leader_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } } }
I20260927 11:17:17.028995 23221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [sys.catalog]: This master's current role is: LEADER
W20260927 11:17:17.029270 23232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:17:17.029325 23232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:17:17.029383 22983 master_runner.cc:186] Error getting master registration for 127.0.98.61:42539: OK, instance_id { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" instance_seqno: 1790507836817804 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 11:17:17.032132 22983 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 11:17:17.035578 23241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 11:17:17.035753 23241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 11:17:17.037145 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:17.028320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.0.98.62:41131)
0927 11:17:17.028354 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.028356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.028863 (+ 507us) server_negotiation.cc:407] Connection header received
0927 11:17:17.028995 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.028996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.029018 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.029046 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.030397 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:17.031106 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:17.032541 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:17.032853 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:17.032929 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:17.034345 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:17.034346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:17.034346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:17.034353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:17.035984 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:17.036404 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:17.036405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:17.036405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:17.036454 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:17.036638 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:17.036639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:17.036639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:17.036744 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:17.036807 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:17.036995 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 11:17:17.037063 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 11:17:17.038687 23175 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf [term 1 FOLLOWER]: Refusing update from remote peer 35d90a5f4d4047f8b7817f00d9da5a5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:17:17.039124 23238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 11:17:17.040575 23207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35d90a5f4d4047f8b7817f00d9da5a5e. Latest consensus state: current_term: 1 leader_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } } }
I20260927 11:17:17.040689 23207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:17:17.046291 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:17.028839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37312 (local address 127.0.98.60:38241)
0927 11:17:17.028871 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.028873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.034345 (+ 5472us) server_negotiation.cc:407] Connection header received
0927 11:17:17.034434 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.034435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.034444 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.034471 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.036431 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:17.037072 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:17.037946 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:17.038266 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:17.038304 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:17.043052 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:17.043053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:17.043053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:17.043060 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:17.044651 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:17.045161 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:17.045163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:17.045163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:17.045233 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:17.045626 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:17.045627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:17.045627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:17.045766 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:17.045851 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:17.046123 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 11:17:17.046209 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 11:17:17.047972 23221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } } }
I20260927 11:17:17.048048 23221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:17.048141 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:17.028443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33168 (local address 127.0.98.61:42539)
0927 11:17:17.028498 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.028500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.034194 (+ 5694us) server_negotiation.cc:407] Connection header received
0927 11:17:17.034278 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.034280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.034290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.034341 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.035875 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:17.036501 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:17.037809 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:17.038133 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:17.038170 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:17.041589 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:17.041590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:17.041590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:17.041609 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:17.044546 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:17.047049 (+ 2503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:17.047050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:17.047051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:17.047119 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:17.047448 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:17.047449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:17.047449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:17.047571 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:17.047644 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:17.047949 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 11:17:17.048029 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 11:17:17.048655 23244 mvcc.cc:205] Tried to move back new op lower bound from 7333920100507660288 to 7333920100469149696. Current Snapshot: MvccSnapshot[applied={T|T < 7333920100507660288}]
I20260927 11:17:17.048910 23245 mvcc.cc:205] Tried to move back new op lower bound from 7333920100507660288 to 7333920100469149696. Current Snapshot: MvccSnapshot[applied={T|T < 7333920100507660288}]
I20260927 11:17:17.049000 23207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } } }
I20260927 11:17:17.049080 23207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:17:17.050017 23126 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } attempt: 3
I20260927 11:17:17.054394 23241 catalog_manager.cc:1347] Generated new cluster ID: 34f00ea2eddb4308acfd0ae147515d08
I20260927 11:17:17.054445 23241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 11:17:17 dist-test-slave-9hq9 krb5kdc[22970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790507836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 11:17:17.057930 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:17.050267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38241 (local address 127.0.0.1:37324)
0927 11:17:17.050506 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 11:17:17.050515 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 11:17:17.050553 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 11:17:17.050870 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 11:17:17.050870 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 11:17:17.051012 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 11:17:17.051045 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:17.051047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:17.051877 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:17.051878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:17.052668 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 11:17:17.052670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 11:17:17.053119 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 11:17:17.053120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 11:17:17.053185 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:17.053562 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 11:17:17.053566 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 11:17:17.055064 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 11:17:17.056918 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:17.056919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:17.056919 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:17.057107 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:17.057307 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 11:17:17.057307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 11:17:17.057308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 11:17:17.057376 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 11:17:17.057659 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 11:17:17.057660 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 11:17:17.057709 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 11:17:17.057770 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 11:17:17.057846 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223}
I20260927 11:17:17.058030 23204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:17.050700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.0.98.60:38241)
0927 11:17:17.050761 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.050762 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.050769 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:17:17.050785 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.050786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.050794 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.050835 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.051114 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:17.051809 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:17.052756 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:17.053046 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:17.053105 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:17.055388 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 11:17:17.055389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 11:17:17.055389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 11:17:17.055395 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 11:17:17.056832 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:17.057177 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:17.057178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:17.057178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:17.057232 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 11:17:17.057442 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 11:17:17.057443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 11:17:17.057443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 11:17:17.057590 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 11:17:17.057665 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:17.057805 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 11:17:17.057914 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 11:17:17.059520 23126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3: Bootstrap starting.
I20260927 11:17:17.060128 23126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3: Neither blocks nor log segments found. Creating new log.
I20260927 11:17:17.060377 23126 log.cc:824] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3: Log is configured to *not* fsync() on all Append() calls
I20260927 11:17:17.061113 23126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3: No bootstrap required, opened a new log
I20260927 11:17:17.062995 23126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } }
I20260927 11:17:17.063123 23126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ce752816af4f8da7fa4f958b9777a3, State: Initialized, Role: FOLLOWER
I20260927 11:17:17.063194 23126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } }
I20260927 11:17:17.063472 23126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 11:17:17.063535 23100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "66ce752816af4f8da7fa4f958b9777a3"
I20260927 11:17:17.063647 23100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 11:17:17.064512 23100 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35d90a5f4d4047f8b7817f00d9da5a5e in term 1.
I20260927 11:17:17.064790 23099 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3 [term 1 FOLLOWER]: Refusing update from remote peer 35d90a5f4d4047f8b7817f00d9da5a5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 11:17:17.063477 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } } }
I20260927 11:17:17.065639 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:17:17.065655 23238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 11:17:17.065821 23079 tablet.cc:2369] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 11:17:17.066076 23241 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 11:17:17.066744 23241 catalog_manager.cc:1504] Loading token signing keys...
I20260927 11:17:17.066895 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35d90a5f4d4047f8b7817f00d9da5a5e. Latest consensus state: current_term: 1 leader_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } } }
I20260927 11:17:17.066967 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:17:17.067000 23251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } } }
I20260927 11:17:17.067029 23251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 11:17:17.070550 23263 mvcc.cc:205] Tried to move back new op lower bound from 7333920100609814528 to 7333920100469149696. Current Snapshot: MvccSnapshot[applied={T|T < 7333920100507660288}]
W20260927 11:17:17.071018 23262 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 11:17:17.071069 23262 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 11:17:17.077530 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d90a5f4d4047f8b7817f00d9da5a5e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } peers { permanent_uuid: "66ce752816af4f8da7fa4f958b9777a3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42539 } } peers { permanent_uuid: "3ee212dd44434c5cb56988f7f2678bdf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38241 } } }
I20260927 11:17:17.077620 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e [sys.catalog]: This master's current role is: LEADER
I20260927 11:17:17.079187 23241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35d90a5f4d4047f8b7817f00d9da5a5e: Generated new TSK 0
I20260927 11:17:17.079339 23241 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 11:17:17.104437 23055 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:17.102867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33968 (local address 127.0.98.62:41131)
0927 11:17:17.102907 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.102909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.103780 (+ 871us) server_negotiation.cc:407] Connection header received
0927 11:17:17.103902 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.103903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.103913 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.103943 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.104378 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33968: BlockingRecv error: recv got EOF from 127.0.0.1:33968 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 11:17:17.105173 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:17.103242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33176 (local address 127.0.98.61:42539)
0927 11:17:17.103279 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.103281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.104641 (+ 1360us) server_negotiation.cc:407] Connection header received
0927 11:17:17.104737 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.104738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.104748 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.104776 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.105123 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33176: BlockingRecv error: recv got EOF from 127.0.0.1:33176 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 11:17:17.105913 23204 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:17.103514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.0.98.60:38241)
0927 11:17:17.103551 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.103552 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.105405 (+ 1853us) server_negotiation.cc:407] Connection header received
0927 11:17:17.105525 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.105526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.105535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.105565 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.105861 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37332: BlockingRecv error: recv got EOF from 127.0.0.1:37332 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 11:17:17.107986 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:17.107321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.0.98.61:42539)
0927 11:17:17.107364 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.107365 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.107509 (+ 144us) server_negotiation.cc:407] Connection header received
0927 11:17:17.107609 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.107610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.107625 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.107654 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.107946 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33192: BlockingRecv error: recv got EOF from 127.0.0.1:33192 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260927 11:17:17.111006 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:17.110264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33200 (local address 127.0.98.61:42539)
0927 11:17:17.110295 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.110297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.110540 (+ 243us) server_negotiation.cc:407] Connection header received
0927 11:17:17.110633 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.110634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.110644 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.110672 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.110966 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33200: BlockingRecv error: recv got EOF from 127.0.0.1:33200 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 11:17:17.114857 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:17.114142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33212 (local address 127.0.98.61:42539)
0927 11:17:17.114176 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.114178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.114395 (+ 217us) server_negotiation.cc:407] Connection header received
0927 11:17:17.114485 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.114486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.114495 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.114522 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.114822 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33212: BlockingRecv error: recv got EOF from 127.0.0.1:33212 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 11:17:17.120609 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:17.119852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.0.98.61:42539)
0927 11:17:17.119884 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.119886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.120125 (+ 239us) server_negotiation.cc:407] Connection header received
0927 11:17:17.120245 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.120246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.120256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.120284 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.120568 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33222: BlockingRecv error: recv got EOF from 127.0.0.1:33222 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 11:17:17.130450 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:17.129677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.0.98.61:42539)
0927 11:17:17.129708 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.129710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.129958 (+ 248us) server_negotiation.cc:407] Connection header received
0927 11:17:17.130048 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.130049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.130058 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.130110 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.130413 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33228: BlockingRecv error: recv got EOF from 127.0.0.1:33228 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 11:17:17.148543 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:17.147634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.0.98.61:42539)
0927 11:17:17.147675 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.147677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.147936 (+ 259us) server_negotiation.cc:407] Connection header received
0927 11:17:17.148032 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.148033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.148044 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.148091 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.148491 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33236: BlockingRecv error: recv got EOF from 127.0.0.1:33236 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 11:17:17.182534 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:17.181802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.0.98.61:42539)
0927 11:17:17.181858 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.181861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.181988 (+ 127us) server_negotiation.cc:407] Connection header received
0927 11:17:17.182080 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.182081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.182093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.182162 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.182468 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33252: BlockingRecv error: recv got EOF from 127.0.0.1:33252 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 11:17:17.248229 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:17.247545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33258 (local address 127.0.98.61:42539)
0927 11:17:17.247612 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.247614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.247722 (+ 108us) server_negotiation.cc:407] Connection header received
0927 11:17:17.247781 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.247782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.247790 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.247856 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.248158 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33258: BlockingRecv error: recv got EOF from 127.0.0.1:33258 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 11:17:17.378265 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:17.377493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.0.98.61:42539)
0927 11:17:17.377564 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.377566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.377675 (+ 109us) server_negotiation.cc:407] Connection header received
0927 11:17:17.377773 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.377774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.377785 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.377854 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.378202 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33262: BlockingRecv error: recv got EOF from 127.0.0.1:33262 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 11:17:17.635934 23130 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 11:17:17.635403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.0.98.61:42539)
0927 11:17:17.635467 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:17.635469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:17.635479 (+ 10us) server_negotiation.cc:407] Connection header received
0927 11:17:17.635542 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:17.635543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:17.635552 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:17.635611 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 11:17:17.635890 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33274: BlockingRecv error: recv got EOF from 127.0.0.1:33274 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 11:17:17.997648 23218 catalog_manager.cc:1259] Loaded cluster ID: 34f00ea2eddb4308acfd0ae147515d08
I20260927 11:17:17.997712 23218 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf: loading cluster ID for follower catalog manager: success
I20260927 11:17:17.998528 23218 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf: acquiring CA information for follower catalog manager: success
I20260927 11:17:17.998772 23218 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ee212dd44434c5cb56988f7f2678bdf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 11:17:18.071391 23262 catalog_manager.cc:1259] Loaded cluster ID: 34f00ea2eddb4308acfd0ae147515d08
I20260927 11:17:18.071456 23262 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3: loading cluster ID for follower catalog manager: success
I20260927 11:17:18.072341 23262 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3: acquiring CA information for follower catalog manager: success
I20260927 11:17:18.072604 23262 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 66ce752816af4f8da7fa4f958b9777a3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 11:17:18.151595 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:18.149132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.0.98.61:42539)
0927 11:17:18.149243 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:18.149245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:18.149272 (+ 27us) server_negotiation.cc:407] Connection header received
0927 11:17:18.149363 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:18.149364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:18.149408 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:18.149481 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:17:18.149655 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:18.150195 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:18.150921 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:18.151109 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:18.151157 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:18.151286 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:17:18.151389 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:17:18.151444 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:18.151472 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 11:17:18.151489 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":59,"threads_started":1}
I20260927 11:17:18.155349 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:18.153286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.0.98.60:38241)
0927 11:17:18.153393 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:18.153395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:18.153403 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:17:18.153445 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:18.153446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:18.153478 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:18.153546 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:17:18.153622 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:18.154083 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:18.154746 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:18.154928 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:18.154977 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:18.155098 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:17:18.155192 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:17:18.155243 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:18.155277 (+ 34us) server_negotiation.cc:299] Negotiation successful
0927 11:17:18.155293 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":64,"threads_started":1}
I20260927 11:17:18.159797 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:18.157073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33290 (local address 127.0.98.61:42539)
0927 11:17:18.157108 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:18.157110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:18.157117 (+ 7us) server_negotiation.cc:407] Connection header received
0927 11:17:18.157188 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:18.157189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:18.157198 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:18.157254 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:17:18.157386 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:18.158117 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:18.159084 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:18.159309 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:18.159392 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:18.159531 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:17:18.159563 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:17:18.159619 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:18.159709 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 11:17:18.159734 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 11:17:18.160009 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:18.157292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.0.98.60:38241)
0927 11:17:18.157324 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:18.157326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:18.157332 (+ 6us) server_negotiation.cc:407] Connection header received
0927 11:17:18.157352 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:18.157352 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:18.157362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:18.157420 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:17:18.157513 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:18.157974 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:18.159100 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:18.159406 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:18.159465 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:18.159761 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:17:18.159790 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:17:18.159819 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:18.159935 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 11:17:18.159957 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 11:17:18.160068 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 11:17:18.156930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.0.98.62:41131)
0927 11:17:18.157223 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 11:17:18.157225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 11:17:18.157233 (+ 8us) server_negotiation.cc:407] Connection header received
0927 11:17:18.157278 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 11:17:18.157279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 11:17:18.157328 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 11:17:18.157408 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 11:17:18.157582 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:18.158307 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:18.159066 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 11:17:18.159399 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 11:17:18.159465 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 11:17:18.159635 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 11:17:18.159735 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 11:17:18.159797 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 11:17:18.159945 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 11:17:18.159966 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":65,"threads_started":1}
I20260927 11:17:18.166426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 22983
I20260927 11:17:18.171679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 23058
I20260927 11:17:18.176600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskGpH_nd/build/release/bin/kudu with pid 23133
2026-09-27T11:17:18Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2754 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5505 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (262159 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST